Oct 29 22:13:40.873129 ip-10-0-138-2 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 22:13:41.225794 ip-10-0-138-2 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:13:41.225794 ip-10-0-138-2 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 22:13:41.225794 ip-10-0-138-2 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:13:41.225794 ip-10-0-138-2 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:13:41.230078 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.229987    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:41.230078 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230076    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:41.230078 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230081    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230085    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230088    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230091    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230094    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230096    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230099    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230101    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230104    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230107    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230110    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230112    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230115    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230117    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230120    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230122    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230125    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230135    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230138    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230141    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:41.230181 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230143    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230146    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230148    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230151    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230153    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230156    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230158    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230163    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230167    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230170    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230172    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230175    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230178    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230182    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230186    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230191    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230194    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230197    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:41.230669 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230199    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230202    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230204    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230207    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230209    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230212    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230215    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230218    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230220    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230223    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230225    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230228    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230231    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230233    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230236    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230239    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230241    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230244    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230247    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230249    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:41.231121 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230252    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230254    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230257    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230259    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230261    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230264    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230267    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230269    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230273    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230276    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230278    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230281    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230283    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230286    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230288    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230290    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230293    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230295    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230298    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230300    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:41.231595 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230303    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230305    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230308    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230310    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230313    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230316    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230318    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230321    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230323    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230326    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230328    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230331    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230333    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230336    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230338    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230341    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230344    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230347    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230349    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:41.232136 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230352    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:41.232600 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230355    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:41.232600 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230357    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:41.232600 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230360    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:41.232600 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230362    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:41.232600 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230364    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:41.232600 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.230367    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:41.232825 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232799    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:41.232861 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232828    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:41.232861 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232832    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:41.232861 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232835    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:41.232861 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232838    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:41.232861 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232841    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:41.232861 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232843    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:41.232861 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232846    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:41.232861 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232849    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:41.232861 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232851    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:41.232861 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232854    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:41.232861 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232856    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:41.232861 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232865    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232868    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232871    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232874    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232876    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232879    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232881    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232884    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232886    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232889    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232891    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232894    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232896    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232898    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232901    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232903    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232905    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232908    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232910    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232913    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:41.233163 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232916    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232918    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232921    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232923    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232926    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232928    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232931    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232933    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232935    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232939    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232943    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232947    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232950    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232954    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232957    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232959    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232962    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232964    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232966    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:41.233633 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232969    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232971    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232973    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232976    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232979    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232983    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232987    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232990    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232992    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232994    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232997    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.232999    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233003    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233006    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233009    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233011    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233014    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233016    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233019    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:41.234130 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233021    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233024    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233026    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233029    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233031    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233034    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233036    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233040    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233043    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233046    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233048    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233051    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233053    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233056    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233058    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233061    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233063    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233066    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233069    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233071    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:41.234578 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233074    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233078    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233080    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233082    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233085    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233087    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233090    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233092    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233094    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233097    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233099    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233102    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233104    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233106    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233108    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233111    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233616    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233621    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233624    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233627    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:41.235086 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233630    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233633    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233636    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233638    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233641    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233644    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233647    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233649    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233652    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233655    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233658    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233661    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233663    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233665    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233668    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233670    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233673    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233675    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233678    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233680    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:41.235560 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233683    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233685    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233688    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233690    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233693    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233695    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233697    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233700    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233703    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233706    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233709    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233711    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233714    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233719    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233722    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233724    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233727    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233729    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233732    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:41.236087 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233734    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233737    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233739    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233743    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233748    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233750    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233753    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233755    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233758    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233760    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233763    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233765    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233768    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233770    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233773    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233775    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233778    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233780    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233783    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:41.236544 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233785    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233787    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233790    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233793    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233796    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233798    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233801    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233803    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233822    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233826    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233829    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233832    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233835    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233837    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233840    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233843    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233846    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233848    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233851    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:41.237023 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233853    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233855    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233858    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233861    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233863    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233865    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233868    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233870    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233873    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233875    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233878    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233880    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233882    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233885    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233888    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233893    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233896    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233899    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233902    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:41.237529 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233904    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233907    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233911    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233914    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233917    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.233920    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234793    2572 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234804    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234826    2572 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234832    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234837    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234840    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234848    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234852    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234856    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234859    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234862    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234866    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234869    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234872    2572 flags.go:64] FLAG: --cgroup-root=""
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234875    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234878    2572 flags.go:64] FLAG: --client-ca-file=""
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234881    2572 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234884    2572 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 22:13:41.238262 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234889    2572 flags.go:64] FLAG: --cluster-domain=""
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234892    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234895    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234898    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234902    2572 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234906    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234909    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234912    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234915    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234918    2572 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234921    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234924    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234928    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234931    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234935    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234938    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234941    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234944    2572 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234947    2572 flags.go:64] FLAG: --enable-server="true"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234950    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234955    2572 flags.go:64] FLAG: --event-burst="100"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234958    2572 flags.go:64] FLAG: --event-qps="50"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234962    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234965    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 22:13:41.239025 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234968    2572 flags.go:64] FLAG: --eviction-hard=""
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234973    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234976    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234978    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234981    2572 flags.go:64] FLAG: --eviction-soft=""
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234984    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234987    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234990    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234993    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234996    2572 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.234999    2572 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235002    2572 flags.go:64] FLAG: --feature-gates=""
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235006    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235009    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235012    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235016    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235019    2572 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235022    2572 flags.go:64] FLAG: --help="false"
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235025    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235028    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235031    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235034    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235037    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 22:13:41.239655 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235041    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235043    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235046    2572 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235049    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235053    2572 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235056    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235059    2572 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235061    2572 flags.go:64] FLAG: --kube-reserved=""
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235064    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235067    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235070    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235073    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235075    2572 flags.go:64] FLAG: --lock-file=""
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235078    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235081    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235084    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235091    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235094    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235097    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235100    2572 flags.go:64] FLAG: --logging-format="text"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235103    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235107    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235110    2572 flags.go:64] FLAG: --manifest-url=""
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235112    2572 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235117    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 22:13:41.240239 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235120    2572 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235124    2572 flags.go:64] FLAG: --max-pods="110"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235127    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235130    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235133    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235136    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235139    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235142    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235146    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235153    2572 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235156    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235159    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235162    2572 flags.go:64] FLAG: --pod-cidr=""
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235165    2572 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235168    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235171    2572 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235174    2572 flags.go:64] FLAG: --port="10250"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235177    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235179    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b5bb37fcb252ca55"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235182    2572 flags.go:64] FLAG: --qos-reserved=""
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235185    2572 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235188    2572 flags.go:64] FLAG: --register-node="true"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235192    2572 flags.go:64] FLAG: --register-with-taints=""
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235198    2572 flags.go:64] FLAG: --registry-burst="10"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235246    2572 flags.go:64] FLAG: --registry-qps="5"
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235253    2572 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 22:13:41.240854 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235256    2572 flags.go:64] FLAG: --reserved-memory=""
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235261    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235264    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235267    2572 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235270    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235273    2572 flags.go:64] FLAG: --runonce="false"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235275    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235279    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235282    2572 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235284    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235287    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235290    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235293    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235297    2572 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235300    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235303    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235306    2572 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235309    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235312    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235315    2572 flags.go:64] FLAG: --system-cgroups=""
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235318    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235323    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235326    2572 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235329    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235333    2572 flags.go:64] FLAG: --tls-min-version=""
Oct 29 22:13:41.241456 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235336    2572 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235339    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235342    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235346    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235350    2572 flags.go:64] FLAG: --v="2"
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235355    2572 flags.go:64] FLAG: --version="false"
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235360    2572 flags.go:64] FLAG: --vmodule=""
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235364    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.235368    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236140    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236146    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236150    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236153    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236156    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236159    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236161    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236164    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236166    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236169    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236172    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236174    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:41.242059 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236177    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236180    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236182    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236185    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236187    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236191    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236193    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236195    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236198    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236200    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236203    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236205    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236208    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236210    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236213    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236216    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236218    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236223    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236225    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:41.242614 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236228    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236230    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236232    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236235    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236237    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236240    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236242    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236244    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236247    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236249    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236251    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236254    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236256    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236259    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236261    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236264    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236266    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236269    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236271    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:41.243074 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236274    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236276    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236279    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236281    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236284    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236286    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236289    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236291    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236294    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236297    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236299    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236303    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236305    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236308    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236310    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236313    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236315    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236318    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236320    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236322    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:41.243523 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236325    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236327    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236330    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236332    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236335    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236337    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236340    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236342    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236347    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236351    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236354    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236357    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236359    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236362    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236364    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236366    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236369    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236371    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236374    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236376    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:41.244010 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236379    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:41.244498 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236381    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:41.244498 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236383    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:41.244498 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236387    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:41.244498 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236390    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:41.244498 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236394    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:41.244498 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236397    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:41.244498 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236400    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:41.244498 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236403    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:41.244498 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236405    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:41.244498 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236408    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:41.244498 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236410    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:41.244498 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236412    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:41.244498 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236415    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:41.244498 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236418    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:41.244498 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236420    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:41.244498 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.236451    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236489    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236494    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236496    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236499    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236503    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236505    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236508    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236511    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236514    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236518    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236522    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236526    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236529    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236532    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236534    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236537    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236539    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236542    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236545    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:41.244961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236548    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236551    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236554    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236556    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236559    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236561    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236564    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236567    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236570    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236573    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236576    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236578    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236581    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236583    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236586    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236588    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236591    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236594    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236597    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:41.245438 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236599    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236602    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236604    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236606    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236609    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236611    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236615    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236618    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236620    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236623    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236625    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236628    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236632    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236634    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236636    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236639    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236642    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236644    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236646    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236649    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:41.245912 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236651    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236654    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236656    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236659    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236661    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236663    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236666    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236668    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236671    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236673    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236676    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236678    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236681    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236684    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236686    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236689    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236691    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236694    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236697    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:41.246391 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236700    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236702    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236705    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236707    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236710    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236713    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236716    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236718    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236721    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236723    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236726    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236728    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236731    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236734    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236736    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236739    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236741    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236744    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236746    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:41.246847 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236749    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236751    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236754    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236756    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236760    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236762    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236764    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236767    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236770    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.236772    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.246238    2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.246256    2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.246883    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.246889    2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.246957    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.246963    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.246967    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.246970    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.246973    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:41.247295 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.246976    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.246978    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.246981    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.246983    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.246986    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.246989    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.246991    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.246994    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.246997    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247000    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247003    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247006    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247008    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247011    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247013    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247016    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247019    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247021    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247023    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247026    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:41.247738 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247028    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247032    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247034    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247043    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247047    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247050    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247052    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247055    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247057    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247060    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247062    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247065    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247068    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247070    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247073    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247075    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247077    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247080    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247083    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:41.248275 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247085    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247088    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247090    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247092    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247095    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247098    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247101    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247103    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247106    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247108    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247110    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247113    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247115    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247118    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247120    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247123    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247125    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247128    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247131    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:41.248720 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247134    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247137    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247139    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247142    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247145    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247150    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247153    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247156    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247159    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247161    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247163    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247166    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247168    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247171    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247173    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247175    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247179    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247181    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247184    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247188    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:41.249169 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247190    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247193    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247195    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247198    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247200    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247203    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247206    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247208    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247212    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247217    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247221    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247224    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247227    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247229    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247233    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247235    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247238    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247240    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247243    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:41.249648 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247245    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:41.250108 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247248    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:41.250108 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247250    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:41.250108 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247253    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:41.250108 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.247286    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:13:41.250108 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247426    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:41.250108 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247430    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:41.250108 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247433    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:41.250108 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247436    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:41.250108 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247438    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:41.250108 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247441    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:41.250108 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247444    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:41.250108 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247446    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:41.250108 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247450    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:41.250108 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247452    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:41.250108 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247455    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:41.250108 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247457    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247460    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247462    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247476    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247479    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247482    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247485    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247488    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247491    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247494    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247496    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247499    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247501    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247504    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247506    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247509    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247512    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247514    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247517    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:41.250488 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247519    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247522    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247524    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247527    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247529    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247532    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247534    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247537    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247540    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247543    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247545    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247548    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247551    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247553    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247556    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247558    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247561    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247563    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247566    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:41.250961 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247569    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247571    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247573    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247576    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247578    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247581    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247583    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247586    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247588    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247591    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247593    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247595    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247598    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247601    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247603    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247606    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247608    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247612    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247616    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247619    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:41.251453 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247622    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247625    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247628    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247631    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247633    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247636    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247638    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247641    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247643    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247645    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247648    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247651    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247653    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247656    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247658    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247661    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247663    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247666    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247668    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247671    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:41.251970 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247673    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:41.252449 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247676    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:41.252449 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247679    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:41.252449 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247683    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:41.252449 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247685    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:41.252449 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247688    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:41.252449 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247691    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:41.252449 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247693    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:41.252449 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247695    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:41.252449 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247698    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:41.252449 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247700    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:41.252449 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247703    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:41.252449 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247705    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:41.252449 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247708    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:41.252449 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247710    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:41.252449 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247727    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:41.252449 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247730    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:41.252841 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.247755    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:13:41.252841 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247790    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:41.252841 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247795    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:41.252841 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247798    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:41.252841 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247801    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:41.252841 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247804    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:41.252841 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247824    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:41.252841 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247829    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:41.252841 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247832    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:41.252841 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247835    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:41.252841 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247838    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:41.252841 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247840    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:41.252841 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247842    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:41.252841 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247845    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:41.252841 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247848    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:41.252841 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247850    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:41.252841 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247853    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247855    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247858    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247860    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247863    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247866    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247868    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247871    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247875    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247879    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247882    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247884    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247887    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247889    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247892    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247895    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247898    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247900    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247903    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:41.253249 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247905    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247907    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247910    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247913    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247923    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247926    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247928    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247931    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247934    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247936    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247939    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247941    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247944    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247946    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247950    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247953    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247956    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247958    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247961    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247963    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:41.253719 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247966    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247968    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247970    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247973    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247976    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247978    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247980    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247983    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247986    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247989    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247991    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247994    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247996    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.247999    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248002    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248004    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248006    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248009    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248012    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248014    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:41.254276 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248016    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248019    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248021    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248023    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248026    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248028    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248031    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248034    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248036    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248038    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248041    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248043    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248045    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248048    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248050    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248052    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248055    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248058    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248060    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:41.254840 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248063    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248065    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248068    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248077    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248080    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248082    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248085    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248087    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248091    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248093    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248096    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248098    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:41.248101    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.248165    2572 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.250408    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.251554    2572 server.go:1014] "Starting client certificate rotation"
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.251653    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.251691    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:13:41.255393 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.253569    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:13:41.255859 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.254703    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:13:41.266793 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.266771    2572 log.go:25] "Validated CRI v1 runtime API"
Oct 29 22:13:41.270210 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.270196    2572 log.go:25] "Validated CRI v1 image API"
Oct 29 22:13:41.271104 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.271077    2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 22:13:41.275281 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.275249    2572 fs.go:135] Filesystem UUIDs: map[52c1ed33-7d7f-4680-9bf8-d4a899aa2f2c:/dev/nvme0n1p4 66777707-1b6e-4df8-8917-8ae6b2af411b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 29 22:13:41.275349 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.275281    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 29 22:13:41.282916 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.282768    2572 manager.go:217] Machine: {Timestamp:2026-10-29 22:13:41.281487888 +0000 UTC m=+0.301762155 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099389 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20990956dce6d87ddc7c4da222c7e7 SystemUUID:ec209909-56dc-e6d8-7ddc-7c4da222c7e7 BootID:45cbb238-5b03-4856-83d8-41c35a7f6e26 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fe:40:f1:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fe:40:f1:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:50:2d:d7:3d:6b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 22:13:41.282916 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.282902    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 22:13:41.283045 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.283034    2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 22:13:41.285951 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.285918    2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 22:13:41.286091 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.285953    2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-2.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 22:13:41.286138 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.286097    2572 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 22:13:41.286138 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.286105    2572 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 22:13:41.286195 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.286183    2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:13:41.286223 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.286209    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:13:41.287170 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.287157    2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 22:13:41.291206 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.291193    2572 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 22:13:41.291337 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.291327    2572 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 22:13:41.292028 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.292017    2572 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 22:13:41.292072 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.292036    2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 22:13:41.292072 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.292056    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 22:13:41.292072 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.292068    2572 kubelet.go:405] "Adding apiserver pod source"
Oct 29 22:13:41.292203 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.292077    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 22:13:41.292489 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.292474    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:13:41.294481 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.294453    2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 22:13:41.299592 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.299580    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:13:41.300399 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.300386    2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 22:13:41.300441 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.300417    2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 22:13:41.300572 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.300562    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 22:13:41.300601 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.300584    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 22:13:41.300601 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.300592    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 22:13:41.300601 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.300600    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 22:13:41.300676 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.300607    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 22:13:41.300676 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.300614    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 22:13:41.301156 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.301147    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 22:13:41.301191 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.301157    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 22:13:41.301191 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.301166    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 22:13:41.301191 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.301173    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 22:13:41.301191 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.301181    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 22:13:41.301191 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.301191    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 22:13:41.302468 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.302458    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 22:13:41.302506 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.302469    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 22:13:41.306337 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.306324    2572 server.go:1263] "Started kubelet"
Oct 29 22:13:41.306451 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.306424    2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 22:13:41.307149 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.307030    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 22:13:41.307243 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.307207    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 22:13:41.307196 ip-10-0-138-2 systemd[1]: Started Kubernetes Kubelet.
Oct 29 22:13:41.309524 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.309503    2572 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 22:13:41.311280 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.311260    2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 22:13:41.312918 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:41.312896    2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 29 22:13:41.315759 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.315734    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 22:13:41.316224 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.316201    2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 22:13:41.316346 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:41.316311    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-2.ec2.internal\" not found"
Oct 29 22:13:41.316851 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.316828    2572 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 22:13:41.317000 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.316853    2572 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 22:13:41.317000 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.316827    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 22:13:41.317000 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.316905    2572 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 22:13:41.317000 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.316913    2572 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 22:13:41.319939 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.319905    2572 factory.go:55] Registering systemd factory
Oct 29 22:13:41.320071 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.320056    2572 factory.go:223] Registration of the systemd container factory successfully
Oct 29 22:13:41.320442 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.320423    2572 factory.go:153] Registering CRI-O factory
Oct 29 22:13:41.320537 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.320447    2572 factory.go:223] Registration of the crio container factory successfully
Oct 29 22:13:41.320537 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.320503    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 22:13:41.320537 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.320521    2572 factory.go:103] Registering Raw factory
Oct 29 22:13:41.320537 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.320537    2572 manager.go:1199] Started watching for new ooms in manager
Oct 29 22:13:41.321593 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.321575    2572 manager.go:319] Starting recovery of all containers
Oct 29 22:13:41.332876 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.332849    2572 manager.go:324] Recovery completed
Oct 29 22:13:41.337110 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.337095    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:41.337559 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.337542    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 22:13:41.337661 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.337643    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6wr5c"
Oct 29 22:13:41.337701 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.337665    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6wr5c"
Oct 29 22:13:41.339300 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.339273    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:41.339300 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.339304    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:41.339458 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.339316    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:41.340171 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.340146    2572 cpu_manager.go:226] "Starting" policy="none"
Oct 29 22:13:41.340171 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.340159    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:41.340309 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.340161    2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 22:13:41.340309 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.340288    2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 22:13:41.340704 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.340687    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-2.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 29 22:13:41.340704 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:41.340682    2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-2.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 29 22:13:41.341008 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.340976    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:41.341103 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.341026    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:41.341103 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.341053    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:41.341598 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:41.340691    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-2.ec2.internal.18e31fc831656954  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-2.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-2.ec2.internal,},FirstTimestamp:2026-10-29 22:13:41.306239316 +0000 UTC m=+0.326513581,LastTimestamp:2026-10-29 22:13:41.306239316 +0000 UTC m=+0.326513581,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-2.ec2.internal,}"
Oct 29 22:13:41.343192 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.343178    2572 policy_none.go:50] "Start"
Oct 29 22:13:41.343243 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.343195    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 22:13:41.343243 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.343206    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 22:13:41.350834 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.350802    2572 policy_none.go:44] "Start"
Oct 29 22:13:41.352655 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:41.352632    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 29 22:13:41.373081 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.372911    2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:13:41.386466 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.386427    2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 22:13:41.390610 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.390589    2572 manager.go:341] "Starting Device Plugin manager"
Oct 29 22:13:41.391144 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:41.391124    2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 22:13:41.391144 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.391141    2572 server.go:87] "Starting device plugin registration server"
Oct 29 22:13:41.391427 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.391415    2572 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 22:13:41.391476 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.391432    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 22:13:41.391545 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.391520    2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 22:13:41.391653 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.391642    2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 22:13:41.391706 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.391656    2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 22:13:41.392344 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:41.392321    2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 22:13:41.392449 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:41.392396    2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-2.ec2.internal\" not found"
Oct 29 22:13:41.401625 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.401603    2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:13:41.442884 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.442852    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 22:13:41.445010 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.444230    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 22:13:41.445010 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.444251    2572 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 22:13:41.445010 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.444269    2572 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 22:13:41.445010 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:41.444303    2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 29 22:13:41.460685 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.460665    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:13:41.492445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.492386    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:41.493396 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.493380    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:41.493508 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.493425    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:41.493508 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.493453    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:41.493508 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.493478    2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.517458 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.517424    2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.545223 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.545181    2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"]
Oct 29 22:13:41.547620 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.547605    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.549533 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.549516    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.578715 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.578695    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.580776 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.580758    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.610260 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.610228    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:13:41.610505 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.610489    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:13:41.617431 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.617409    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.618538 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.618520    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c506ffe79e14398e87cdea51ca1d1f0f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-2.ec2.internal\" (UID: \"c506ffe79e14398e87cdea51ca1d1f0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.618580 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.618545    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4504dba633558f5dac66e017907a0e24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"4504dba633558f5dac66e017907a0e24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.618580 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.618565    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4504dba633558f5dac66e017907a0e24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"4504dba633558f5dac66e017907a0e24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.645592 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.645567    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:13:41.645729 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:41.645618    2572 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.645729 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.645633    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.671167 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.671144    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:13:41.671258 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:41.671195    2572 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-2.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.719496 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.719467    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4504dba633558f5dac66e017907a0e24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"4504dba633558f5dac66e017907a0e24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.719597 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.719500    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4504dba633558f5dac66e017907a0e24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"4504dba633558f5dac66e017907a0e24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.719597 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.719521    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c506ffe79e14398e87cdea51ca1d1f0f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-2.ec2.internal\" (UID: \"c506ffe79e14398e87cdea51ca1d1f0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.719597 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.719576    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c506ffe79e14398e87cdea51ca1d1f0f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-2.ec2.internal\" (UID: \"c506ffe79e14398e87cdea51ca1d1f0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.719597 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.719587    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4504dba633558f5dac66e017907a0e24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"4504dba633558f5dac66e017907a0e24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.719741 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.719588    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4504dba633558f5dac66e017907a0e24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"4504dba633558f5dac66e017907a0e24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.914942 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.914897    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:41.916000 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:41.915983    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Oct 29 22:13:42.251279 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.251192    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 29 22:13:42.251841 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.251376    2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 29 22:13:42.251841 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.251381    2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 29 22:13:42.292233 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.292190    2572 apiserver.go:52] "Watching apiserver"
Oct 29 22:13:42.316463 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.316429    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 29 22:13:42.336023 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.335994    2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 22:13:42.336364 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.336341    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-7q7dr","openshift-multus/network-metrics-daemon-t48mn","openshift-ovn-kubernetes/ovnkube-node-zlnh6","openshift-cluster-node-tuning-operator/tuned-pvswk","openshift-image-registry/node-ca-8bl7p","openshift-network-diagnostics/network-check-target-5pq4b","kube-system/konnectivity-agent-v552v","openshift-dns/node-resolver-jls5s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal","kube-system/global-pull-secret-syncer-p5bdf","kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw","openshift-multus/multus-additional-cni-plugins-bn9z2","openshift-network-operator/iptables-alerter-xfm9m"]
Oct 29 22:13:42.339301 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.339283    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v552v"
Oct 29 22:13:42.341773 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.341749    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.343686 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.343659    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:08:41 +0000 UTC" deadline="2028-07-03 10:06:12.261796734 +0000 UTC"
Oct 29 22:13:42.343686 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.343685    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14699h52m29.91811466s"
Oct 29 22:13:42.344494 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.344479    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.347341 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.347319    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.349503 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.349486    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:13:42.352651 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.352633    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.353933 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.353914    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 22:13:42.354048 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.353937    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-cdnlg\""
Oct 29 22:13:42.357618 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.354409    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 22:13:42.357618 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.355413    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:13:42.357618 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:42.355531    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pq4b" podUID="c03b55ec-8ed6-42d6-895c-2e7213a2ba39"
Oct 29 22:13:42.357618 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.356702    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 22:13:42.359156 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.359132    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:13:42.361390 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.361373    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:42.361490 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:42.361454    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5bdf" podUID="02990faa-cba3-4132-9ca7-fe0f7202bbe5"
Oct 29 22:13:42.363708 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.363690    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:42.363832 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:42.363766    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t48mn" podUID="9b3ec29f-f4e7-4811-83ab-50680e0e60e7"
Oct 29 22:13:42.365001 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.364967    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 22:13:42.365096 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.365050    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xcs9f"
Oct 29 22:13:42.366145 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.366115    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.368866 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.368841    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:13:42.418549 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.418525    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 22:13:42.422504 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422480    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-device-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.422613 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422505    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-run\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.422613 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422522    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-lib-modules\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.422613 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422545    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-ovnkube-config\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.422613 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422588    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-ovnkube-script-lib\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.422613 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422605    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3c107fc-072e-45ce-b4cd-acbe1284be88-tmp\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.422764 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422656    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02990faa-cba3-4132-9ca7-fe0f7202bbe5-kubelet-config\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:42.422764 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422692    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:42.422764 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422712    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc8f498e-4039-43ef-91f7-036179511f01-iptables-alerter-script\") pod \"iptables-alerter-xfm9m\" (UID: \"cc8f498e-4039-43ef-91f7-036179511f01\") " pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:13:42.422764 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422738    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-run-netns\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.422764 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422760    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.422964 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422776    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsljd\" (UniqueName: \"kubernetes.io/projected/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-kube-api-access-vsljd\") pod \"node-ca-8bl7p\" (UID: \"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55\") " pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:13:42.422964 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422793    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.422964 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422841    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-socket-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.422964 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422872    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysctl-conf\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.422964 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422901    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml2q5\" (UniqueName: \"kubernetes.io/projected/198638bc-ff2e-4c81-a599-616e7b015749-kube-api-access-ml2q5\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.422964 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422945    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-conf-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.423131 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.422967    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.423131 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423009    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-var-lib-kubelet\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.423131 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423025    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-tuned\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.423131 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423057    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8cpn\" (UniqueName: \"kubernetes.io/projected/f3c107fc-072e-45ce-b4cd-acbe1284be88-kube-api-access-m8cpn\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.423131 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423087    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-system-cni-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.423131 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423105    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-k8s-cni-cncf-io\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.423291 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423131    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-etc-kubernetes\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.423291 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423203    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-serviceca\") pod \"node-ca-8bl7p\" (UID: \"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55\") " pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:13:42.423291 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423223    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2bf4\" (UniqueName: \"kubernetes.io/projected/23921851-35c8-450c-a184-fe5ed0b1c6a7-kube-api-access-x2bf4\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.423291 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423239    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:42.423291 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423253    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37a98266-fad3-440d-95d4-85de9ad6f476-cni-binary-copy\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.423291 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423274    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-etc-selinux\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.423453 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423308    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n27db\" (UniqueName: \"kubernetes.io/projected/37a98266-fad3-440d-95d4-85de9ad6f476-kube-api-access-n27db\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.423453 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423330    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21-agent-certs\") pod \"konnectivity-agent-v552v\" (UID: \"1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21\") " pod="kube-system/konnectivity-agent-v552v"
Oct 29 22:13:42.423453 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423347    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc8f498e-4039-43ef-91f7-036179511f01-host-slash\") pod \"iptables-alerter-xfm9m\" (UID: \"cc8f498e-4039-43ef-91f7-036179511f01\") " pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:13:42.423453 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423373    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffhsv\" (UniqueName: \"kubernetes.io/projected/cc8f498e-4039-43ef-91f7-036179511f01-kube-api-access-ffhsv\") pod \"iptables-alerter-xfm9m\" (UID: \"cc8f498e-4039-43ef-91f7-036179511f01\") " pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:13:42.423453 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423402    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-socket-dir-parent\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.423453 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423421    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-registration-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.423453 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423439    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvr8w\" (UniqueName: \"kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w\") pod \"network-check-target-5pq4b\" (UID: \"c03b55ec-8ed6-42d6-895c-2e7213a2ba39\") " pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:13:42.423762 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423463    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-system-cni-dir\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.423762 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423484    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.423762 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423530    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-sys\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.423762 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423576    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-host\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.423762 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423607    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-hosts-file\") pod \"node-resolver-jls5s\" (UID: \"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de\") " pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:13:42.423762 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423624    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-log-socket\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.423762 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423645    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-cni-bin\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.423762 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423667    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-kubelet\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.423762 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423684    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-systemd\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.423762 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423697    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-node-log\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.423762 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423720    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-env-overrides\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.423762 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423760    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-kubelet\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.424338 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423789    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-netns\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.424338 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423825    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-hostroot\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.424338 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423848    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-tmp-dir\") pod \"node-resolver-jls5s\" (UID: \"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de\") " pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:13:42.424338 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423862    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-cni-bin\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.424338 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423875    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-cni-netd\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.424338 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423888    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/198638bc-ff2e-4c81-a599-616e7b015749-ovn-node-metrics-cert\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.424338 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423906    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-sys-fs\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.424338 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423945    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-var-lib-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.424338 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423974    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37a98266-fad3-440d-95d4-85de9ad6f476-multus-daemon-config\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.424338 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.423997    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21-konnectivity-ca\") pod \"konnectivity-agent-v552v\" (UID: \"1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21\") " pod="kube-system/konnectivity-agent-v552v"
Oct 29 22:13:42.424338 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424029    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-systemd\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.424338 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424053    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-slash\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.424338 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424075    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-multus-certs\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.424338 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424098    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.424338 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424122    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.424338 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424138    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysctl-d\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.425078 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424154    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbch6\" (UniqueName: \"kubernetes.io/projected/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-kube-api-access-pbch6\") pod \"node-resolver-jls5s\" (UID: \"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de\") " pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:13:42.425078 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424195    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02990faa-cba3-4132-9ca7-fe0f7202bbe5-dbus\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:42.425078 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424215    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-cnibin\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.425078 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424229    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-ovn\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.425078 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424243    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-run-ovn-kubernetes\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.425078 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424266    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsrvw\" (UniqueName: \"kubernetes.io/projected/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-kube-api-access-xsrvw\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:42.425078 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424298    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-systemd-units\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.425078 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424318    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-cnibin\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.425078 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424333    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-kubernetes\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.425078 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424367    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.425078 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424385    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-cni-multus\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.425078 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424401    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxlcz\" (UniqueName: \"kubernetes.io/projected/486bbe97-6300-49e9-a0f3-c9d316333048-kube-api-access-lxlcz\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.425078 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424415    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-modprobe-d\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.425078 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424429    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysconfig\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.425078 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424447    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-etc-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.425078 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424460    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-cni-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.425519 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.424472    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-host\") pod \"node-ca-8bl7p\" (UID: \"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55\") " pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:13:42.454920 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.454892    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:13:42.455086 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.455047    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 22:13:42.455086 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.455052    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 22:13:42.455086 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.455081    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 22:13:42.455226 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.455124    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f584j\""
Oct 29 22:13:42.455450 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.455410    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 22:13:42.455587 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.455450    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 22:13:42.455587 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.455466    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 22:13:42.455587 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.455541    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 22:13:42.455587 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.455558    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 22:13:42.455753 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.455597    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kr68l\""
Oct 29 22:13:42.455753 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.455598    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 22:13:42.455896 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.455756    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mwjl8\""
Oct 29 22:13:42.455954 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.455924    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 22:13:42.456212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.456171    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 22:13:42.456332 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.456241    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bjh7c\""
Oct 29 22:13:42.456332 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.456245    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bwm9d\""
Oct 29 22:13:42.456332 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.456303    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 22:13:42.456624 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.456609    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 22:13:42.456779 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.456749    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8zjwp\""
Oct 29 22:13:42.467244 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.467215    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 22:13:42.467244 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.467215    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fz5pc\""
Oct 29 22:13:42.467380 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.467329    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 22:13:42.467380 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.467346    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 22:13:42.467453 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.467404    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 22:13:42.467805 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.467789    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 22:13:42.467896 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.467822    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 22:13:42.467896 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.467841    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 22:13:42.467896 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.467862    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 22:13:42.469463 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.468713    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:13:42.469463 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.468781    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p5dkc\""
Oct 29 22:13:42.469463 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.468864    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 22:13:42.470129 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:42.470084    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc506ffe79e14398e87cdea51ca1d1f0f.slice/crio-09ec796e175c68a7ecf208e3fd25136253c240fb603980b9d7cf51f6c295cad2 WatchSource:0}: Error finding container 09ec796e175c68a7ecf208e3fd25136253c240fb603980b9d7cf51f6c295cad2: Status 404 returned error can't find the container with id 09ec796e175c68a7ecf208e3fd25136253c240fb603980b9d7cf51f6c295cad2
Oct 29 22:13:42.482192 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.482176    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:13:42.494389 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:42.494361    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4504dba633558f5dac66e017907a0e24.slice/crio-e075d51ca14e4f7237ba867d6845ddcd67c963a12ee720c3c8786c47ebcebac9 WatchSource:0}: Error finding container e075d51ca14e4f7237ba867d6845ddcd67c963a12ee720c3c8786c47ebcebac9: Status 404 returned error can't find the container with id e075d51ca14e4f7237ba867d6845ddcd67c963a12ee720c3c8786c47ebcebac9
Oct 29 22:13:42.524871 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.524775    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-cni-bin\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.524871 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.524822    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-kubelet\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.524871 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.524840    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-systemd\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.524871 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.524854    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-node-log\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.524871 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.524868    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-env-overrides\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.524881    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-kubelet\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.524888    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-cni-bin\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.524925    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-node-log\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.524895    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-netns\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.524965    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-netns\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.524970    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-kubelet\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525001    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-hostroot\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.524946    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-systemd\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.524974    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-hostroot\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.524888    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-kubelet\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525053    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-tmp-dir\") pod \"node-resolver-jls5s\" (UID: \"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de\") " pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525083    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-cni-bin\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525109    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-cni-netd\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525142    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-cni-bin\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525158    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-cni-netd\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525161    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/198638bc-ff2e-4c81-a599-616e7b015749-ovn-node-metrics-cert\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525198    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-sys-fs\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.525247 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525223    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-var-lib-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.525824 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525247    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37a98266-fad3-440d-95d4-85de9ad6f476-multus-daemon-config\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.525824 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525305    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-sys-fs\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.525824 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525305    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-var-lib-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.525824 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525336    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21-konnectivity-ca\") pod \"konnectivity-agent-v552v\" (UID: \"1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21\") " pod="kube-system/konnectivity-agent-v552v"
Oct 29 22:13:42.525824 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525361    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-systemd\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.525824 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525376    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-slash\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.525824 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525392    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-multus-certs\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.525824 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525418    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.525824 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525442    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-systemd\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.525824 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525443    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.525824 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525461    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-slash\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.525824 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525476    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysctl-d\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.525824 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525500    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbch6\" (UniqueName: \"kubernetes.io/projected/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-kube-api-access-pbch6\") pod \"node-resolver-jls5s\" (UID: \"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de\") " pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:13:42.525824 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525517    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02990faa-cba3-4132-9ca7-fe0f7202bbe5-dbus\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:42.525824 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525531    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-cnibin\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.525824 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525544    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-ovn\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.525824 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525560    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-run-ovn-kubernetes\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.526630 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525576    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsrvw\" (UniqueName: \"kubernetes.io/projected/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-kube-api-access-xsrvw\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:42.526630 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525588    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysctl-d\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.526630 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525592    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-systemd-units\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.526630 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525606    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-cnibin\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.526630 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525616    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-cnibin\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.526630 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525619    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-kubernetes\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.526630 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525636    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.526630 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525655    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-cni-multus\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.526630 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525500    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-multus-certs\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.526630 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525670    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlcz\" (UniqueName: \"kubernetes.io/projected/486bbe97-6300-49e9-a0f3-c9d316333048-kube-api-access-lxlcz\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.526630 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525707    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-modprobe-d\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.526630 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525737    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysconfig\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.526630 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525762    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-etc-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.526630 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525785    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-cni-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.526630 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525802    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-host\") pod \"node-ca-8bl7p\" (UID: \"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55\") " pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:13:42.526630 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525868    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-device-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.526630 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525862    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-ovn\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525893    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-run\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525902    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysconfig\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525916    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-lib-modules\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525922    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-run-ovn-kubernetes\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525950    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-ovnkube-config\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525977    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-ovnkube-script-lib\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525994    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-modprobe-d\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.525998    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3c107fc-072e-45ce-b4cd-acbe1284be88-tmp\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526022    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02990faa-cba3-4132-9ca7-fe0f7202bbe5-kubelet-config\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526037    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-env-overrides\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526045    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526053    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-tmp-dir\") pod \"node-resolver-jls5s\" (UID: \"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de\") " pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526063    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02990faa-cba3-4132-9ca7-fe0f7202bbe5-dbus\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526058    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-run\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526076    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc8f498e-4039-43ef-91f7-036179511f01-iptables-alerter-script\") pod \"iptables-alerter-xfm9m\" (UID: \"cc8f498e-4039-43ef-91f7-036179511f01\") " pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526088    2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526111    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-cni-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.527445 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526133    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-run-netns\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.528326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526158    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-lib-modules\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.528326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526164    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.528326 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:42.526189    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:42.528326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526198    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsljd\" (UniqueName: \"kubernetes.io/projected/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-kube-api-access-vsljd\") pod \"node-ca-8bl7p\" (UID: \"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55\") " pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:13:42.528326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526213    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-cnibin\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.528326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526228    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.528326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526275    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-kubernetes\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.528326 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:42.526297    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs podName:9b3ec29f-f4e7-4811-83ab-50680e0e60e7 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:43.0262405 +0000 UTC m=+2.046514772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs") pod "network-metrics-daemon-t48mn" (UID: "9b3ec29f-f4e7-4811-83ab-50680e0e60e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:42.528326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526327    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.528326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526365    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-cni-multus\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.528326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526415    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-socket-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.528326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526469    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysctl-conf\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.528326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526495    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml2q5\" (UniqueName: \"kubernetes.io/projected/198638bc-ff2e-4c81-a599-616e7b015749-kube-api-access-ml2q5\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.528326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526498    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.528326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526170    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02990faa-cba3-4132-9ca7-fe0f7202bbe5-kubelet-config\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:42.528326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526515    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-conf-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.528326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526543    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-conf-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.529212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526550    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.529212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526564    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-device-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.529212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526592    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-var-lib-kubelet\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.529212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526616    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-tuned\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.529212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526638    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-socket-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.529212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526640    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8cpn\" (UniqueName: \"kubernetes.io/projected/f3c107fc-072e-45ce-b4cd-acbe1284be88-kube-api-access-m8cpn\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.529212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526666    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc8f498e-4039-43ef-91f7-036179511f01-iptables-alerter-script\") pod \"iptables-alerter-xfm9m\" (UID: \"cc8f498e-4039-43ef-91f7-036179511f01\") " pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:13:42.529212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526671    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-system-cni-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.529212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526703    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-system-cni-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.529212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526723    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-k8s-cni-cncf-io\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.529212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526726    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-ovnkube-script-lib\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.529212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526751    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-etc-kubernetes\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.529212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526778    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-serviceca\") pod \"node-ca-8bl7p\" (UID: \"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55\") " pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:13:42.529212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526791    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-host\") pod \"node-ca-8bl7p\" (UID: \"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55\") " pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:13:42.529212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526804    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2bf4\" (UniqueName: \"kubernetes.io/projected/23921851-35c8-450c-a184-fe5ed0b1c6a7-kube-api-access-x2bf4\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.529212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526847    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysctl-conf\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.529212 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526856    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-etc-kubernetes\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526857    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526897    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37a98266-fad3-440d-95d4-85de9ad6f476-cni-binary-copy\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526919    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-etc-selinux\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:42.526925    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526942    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n27db\" (UniqueName: \"kubernetes.io/projected/37a98266-fad3-440d-95d4-85de9ad6f476-kube-api-access-n27db\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526965    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21-agent-certs\") pod \"konnectivity-agent-v552v\" (UID: \"1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21\") " pod="kube-system/konnectivity-agent-v552v"
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.526984    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc8f498e-4039-43ef-91f7-036179511f01-host-slash\") pod \"iptables-alerter-xfm9m\" (UID: \"cc8f498e-4039-43ef-91f7-036179511f01\") " pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527004    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffhsv\" (UniqueName: \"kubernetes.io/projected/cc8f498e-4039-43ef-91f7-036179511f01-kube-api-access-ffhsv\") pod \"iptables-alerter-xfm9m\" (UID: \"cc8f498e-4039-43ef-91f7-036179511f01\") " pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527023    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-socket-dir-parent\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527062    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-registration-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527092    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvr8w\" (UniqueName: \"kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w\") pod \"network-check-target-5pq4b\" (UID: \"c03b55ec-8ed6-42d6-895c-2e7213a2ba39\") " pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527102    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-ovnkube-config\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527114    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-system-cni-dir\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527128    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-etc-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527134    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527165    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-sys\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527178    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.529929 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527179    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-var-lib-kubelet\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.530467 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527189    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-host\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.530467 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527228    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-k8s-cni-cncf-io\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.530467 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527233    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-hosts-file\") pod \"node-resolver-jls5s\" (UID: \"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de\") " pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:13:42.530467 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527259    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-log-socket\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.530467 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527350    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-serviceca\") pod \"node-ca-8bl7p\" (UID: \"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55\") " pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:13:42.530467 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527322    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-etc-selinux\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.530467 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527415    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-host\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.530467 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527434    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-run-netns\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.530467 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527456    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-systemd-units\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.530467 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:42.527533    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret podName:02990faa-cba3-4132-9ca7-fe0f7202bbe5 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:43.027516063 +0000 UTC m=+2.047790324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret") pod "global-pull-secret-syncer-p5bdf" (UID: "02990faa-cba3-4132-9ca7-fe0f7202bbe5") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:42.530467 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527672    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-log-socket\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.530467 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527725    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-hosts-file\") pod \"node-resolver-jls5s\" (UID: \"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de\") " pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:13:42.530467 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.527766    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-socket-dir-parent\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.530467 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.528026    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.530467 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.528049    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-registration-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.530467 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.528095    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-sys\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.530467 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.528127    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-system-cni-dir\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.530952 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.528152    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37a98266-fad3-440d-95d4-85de9ad6f476-multus-daemon-config\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.530952 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.528199    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.530952 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.528386    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc8f498e-4039-43ef-91f7-036179511f01-host-slash\") pod \"iptables-alerter-xfm9m\" (UID: \"cc8f498e-4039-43ef-91f7-036179511f01\") " pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:13:42.530952 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.529004    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21-konnectivity-ca\") pod \"konnectivity-agent-v552v\" (UID: \"1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21\") " pod="kube-system/konnectivity-agent-v552v"
Oct 29 22:13:42.530952 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.529136    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.530952 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.529192    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.530952 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.530377    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-tuned\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.530952 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.530418    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3c107fc-072e-45ce-b4cd-acbe1284be88-tmp\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.530952 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.530422    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37a98266-fad3-440d-95d4-85de9ad6f476-cni-binary-copy\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.530952 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.530453    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/198638bc-ff2e-4c81-a599-616e7b015749-ovn-node-metrics-cert\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.531216 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.530966    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21-agent-certs\") pod \"konnectivity-agent-v552v\" (UID: \"1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21\") " pod="kube-system/konnectivity-agent-v552v"
Oct 29 22:13:42.546333 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:42.546220    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:42.546333 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:42.546324    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:42.546333 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:42.546340    2572 projected.go:196] Error preparing data for projected volume kube-api-access-bvr8w for pod openshift-network-diagnostics/network-check-target-5pq4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:42.546562 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:42.546415    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w podName:c03b55ec-8ed6-42d6-895c-2e7213a2ba39 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:43.046394071 +0000 UTC m=+2.066668340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bvr8w" (UniqueName: "kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w") pod "network-check-target-5pq4b" (UID: "c03b55ec-8ed6-42d6-895c-2e7213a2ba39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:42.547676 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.547646    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbch6\" (UniqueName: \"kubernetes.io/projected/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-kube-api-access-pbch6\") pod \"node-resolver-jls5s\" (UID: \"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de\") " pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:13:42.547782 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.547719    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlcz\" (UniqueName: \"kubernetes.io/projected/486bbe97-6300-49e9-a0f3-c9d316333048-kube-api-access-lxlcz\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.547973 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.547957    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsrvw\" (UniqueName: \"kubernetes.io/projected/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-kube-api-access-xsrvw\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:42.548235 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.548214    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffhsv\" (UniqueName: \"kubernetes.io/projected/cc8f498e-4039-43ef-91f7-036179511f01-kube-api-access-ffhsv\") pod \"iptables-alerter-xfm9m\" (UID: \"cc8f498e-4039-43ef-91f7-036179511f01\") " pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:13:42.549517 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.549501    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsljd\" (UniqueName: \"kubernetes.io/projected/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-kube-api-access-vsljd\") pod \"node-ca-8bl7p\" (UID: \"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55\") " pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:13:42.550325 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.550301    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8cpn\" (UniqueName: \"kubernetes.io/projected/f3c107fc-072e-45ce-b4cd-acbe1284be88-kube-api-access-m8cpn\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.550401 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.550312    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml2q5\" (UniqueName: \"kubernetes.io/projected/198638bc-ff2e-4c81-a599-616e7b015749-kube-api-access-ml2q5\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.552011 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.551989    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n27db\" (UniqueName: \"kubernetes.io/projected/37a98266-fad3-440d-95d4-85de9ad6f476-kube-api-access-n27db\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.552084 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.552019    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2bf4\" (UniqueName: \"kubernetes.io/projected/23921851-35c8-450c-a184-fe5ed0b1c6a7-kube-api-access-x2bf4\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.580109 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.580076    2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:13:42.670667 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.670633    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v552v"
Oct 29 22:13:42.681341 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:42.681313    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6d3ad4_9eca_4ff5_913b_2eb405f4bb21.slice/crio-7b9a07980d4585ffee63d09c704ea6c68527a26286578c28d44f244aaf634d12 WatchSource:0}: Error finding container 7b9a07980d4585ffee63d09c704ea6c68527a26286578c28d44f244aaf634d12: Status 404 returned error can't find the container with id 7b9a07980d4585ffee63d09c704ea6c68527a26286578c28d44f244aaf634d12
Oct 29 22:13:42.687614 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.687594    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:13:42.697650 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:42.697570    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486bbe97_6300_49e9_a0f3_c9d316333048.slice/crio-4750ae6c337a571a725edd2562a3517a062504a3ade721861ff54c1364a5675a WatchSource:0}: Error finding container 4750ae6c337a571a725edd2562a3517a062504a3ade721861ff54c1364a5675a: Status 404 returned error can't find the container with id 4750ae6c337a571a725edd2562a3517a062504a3ade721861ff54c1364a5675a
Oct 29 22:13:42.707969 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.707946    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:13:42.710643 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.710624    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7q7dr"
Oct 29 22:13:42.717484 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:42.717453    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198638bc_ff2e_4c81_a599_616e7b015749.slice/crio-10b41b6adef4be966c582cdc48e40fadb28dd646302d456467848faed34ec445 WatchSource:0}: Error finding container 10b41b6adef4be966c582cdc48e40fadb28dd646302d456467848faed34ec445: Status 404 returned error can't find the container with id 10b41b6adef4be966c582cdc48e40fadb28dd646302d456467848faed34ec445
Oct 29 22:13:42.717899 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.717876    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:13:42.720368 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:42.720343    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a98266_fad3_440d_95d4_85de9ad6f476.slice/crio-0de9b0e763875276bbefb78c429db0f6b6a1d9165ceba2b1e61e6841cf2b46c5 WatchSource:0}: Error finding container 0de9b0e763875276bbefb78c429db0f6b6a1d9165ceba2b1e61e6841cf2b46c5: Status 404 returned error can't find the container with id 0de9b0e763875276bbefb78c429db0f6b6a1d9165ceba2b1e61e6841cf2b46c5
Oct 29 22:13:42.729406 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:42.729382    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51b17e1_8c9b_44cb_b0dc_aed1c4136a55.slice/crio-5e00e456a78e0e6d254bc031812ca35187fdb18b65303fe01df2736f70ebcb99 WatchSource:0}: Error finding container 5e00e456a78e0e6d254bc031812ca35187fdb18b65303fe01df2736f70ebcb99: Status 404 returned error can't find the container with id 5e00e456a78e0e6d254bc031812ca35187fdb18b65303fe01df2736f70ebcb99
Oct 29 22:13:42.738620 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.738599    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:13:42.743870 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.743848    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:13:42.748161 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:42.748138    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c107fc_072e_45ce_b4cd_acbe1284be88.slice/crio-812136f19d236521b0d7ad1f159dc08dcb7cccaa6850c0e91d3918dcd61cf519 WatchSource:0}: Error finding container 812136f19d236521b0d7ad1f159dc08dcb7cccaa6850c0e91d3918dcd61cf519: Status 404 returned error can't find the container with id 812136f19d236521b0d7ad1f159dc08dcb7cccaa6850c0e91d3918dcd61cf519
Oct 29 22:13:42.753787 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:42.753765    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66f6cc5_c6c7_476c_96e9_cd1fb85a78de.slice/crio-408002ab0d76e5b8292d2f62e2ebbbf8b28d0c79d00208c735d954af57dc27b5 WatchSource:0}: Error finding container 408002ab0d76e5b8292d2f62e2ebbbf8b28d0c79d00208c735d954af57dc27b5: Status 404 returned error can't find the container with id 408002ab0d76e5b8292d2f62e2ebbbf8b28d0c79d00208c735d954af57dc27b5
Oct 29 22:13:42.781271 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.781192    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:13:42.783349 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:42.783329    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:13:42.792146 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:42.792120    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23921851_35c8_450c_a184_fe5ed0b1c6a7.slice/crio-044bc26e92f3f0b48c48ddfb35af3edabddcd9e052b53d32258b34a2bcf45694 WatchSource:0}: Error finding container 044bc26e92f3f0b48c48ddfb35af3edabddcd9e052b53d32258b34a2bcf45694: Status 404 returned error can't find the container with id 044bc26e92f3f0b48c48ddfb35af3edabddcd9e052b53d32258b34a2bcf45694
Oct 29 22:13:42.792697 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:13:42.792671    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8f498e_4039_43ef_91f7_036179511f01.slice/crio-c7dc3bc4c972c100a2074f4299f4962f9febc7bc6a8bdb9458e79686fd5c0742 WatchSource:0}: Error finding container c7dc3bc4c972c100a2074f4299f4962f9febc7bc6a8bdb9458e79686fd5c0742: Status 404 returned error can't find the container with id c7dc3bc4c972c100a2074f4299f4962f9febc7bc6a8bdb9458e79686fd5c0742
Oct 29 22:13:43.032865 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.032250    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:43.032865 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.032301    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:43.032865 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:43.032417    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:43.032865 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:43.032473    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret podName:02990faa-cba3-4132-9ca7-fe0f7202bbe5 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:44.032458868 +0000 UTC m=+3.052733121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret") pod "global-pull-secret-syncer-p5bdf" (UID: "02990faa-cba3-4132-9ca7-fe0f7202bbe5") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:43.032865 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:43.032521    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:43.032865 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:43.032563    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs podName:9b3ec29f-f4e7-4811-83ab-50680e0e60e7 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:44.03254897 +0000 UTC m=+3.052823240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs") pod "network-metrics-daemon-t48mn" (UID: "9b3ec29f-f4e7-4811-83ab-50680e0e60e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:43.108472 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.108414    2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:13:43.133138 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.133099    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvr8w\" (UniqueName: \"kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w\") pod \"network-check-target-5pq4b\" (UID: \"c03b55ec-8ed6-42d6-895c-2e7213a2ba39\") " pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:13:43.133363 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:43.133339    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:43.133424 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:43.133373    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:43.133424 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:43.133388    2572 projected.go:196] Error preparing data for projected volume kube-api-access-bvr8w for pod openshift-network-diagnostics/network-check-target-5pq4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:43.133532 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:43.133457    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w podName:c03b55ec-8ed6-42d6-895c-2e7213a2ba39 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:44.133436332 +0000 UTC m=+3.153710621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bvr8w" (UniqueName: "kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w") pod "network-check-target-5pq4b" (UID: "c03b55ec-8ed6-42d6-895c-2e7213a2ba39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:43.370222 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.370089    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:08:42 +0000 UTC" deadline="2028-07-12 03:52:03.307683172 +0000 UTC"
Oct 29 22:13:43.370222 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.370159    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14909h38m19.937528699s"
Oct 29 22:13:43.465430 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.465397    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:43.465621 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:43.465585    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t48mn" podUID="9b3ec29f-f4e7-4811-83ab-50680e0e60e7"
Oct 29 22:13:43.491321 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.491253    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jls5s" event={"ID":"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de","Type":"ContainerStarted","Data":"408002ab0d76e5b8292d2f62e2ebbbf8b28d0c79d00208c735d954af57dc27b5"}
Oct 29 22:13:43.516586 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.516541    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8bl7p" event={"ID":"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55","Type":"ContainerStarted","Data":"5e00e456a78e0e6d254bc031812ca35187fdb18b65303fe01df2736f70ebcb99"}
Oct 29 22:13:43.524981 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.524940    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7q7dr" event={"ID":"37a98266-fad3-440d-95d4-85de9ad6f476","Type":"ContainerStarted","Data":"0de9b0e763875276bbefb78c429db0f6b6a1d9165ceba2b1e61e6841cf2b46c5"}
Oct 29 22:13:43.532856 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.532811    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw" event={"ID":"486bbe97-6300-49e9-a0f3-c9d316333048","Type":"ContainerStarted","Data":"4750ae6c337a571a725edd2562a3517a062504a3ade721861ff54c1364a5675a"}
Oct 29 22:13:43.542832 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.542774    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal" event={"ID":"c506ffe79e14398e87cdea51ca1d1f0f","Type":"ContainerStarted","Data":"09ec796e175c68a7ecf208e3fd25136253c240fb603980b9d7cf51f6c295cad2"}
Oct 29 22:13:43.547086 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.547042    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xfm9m" event={"ID":"cc8f498e-4039-43ef-91f7-036179511f01","Type":"ContainerStarted","Data":"c7dc3bc4c972c100a2074f4299f4962f9febc7bc6a8bdb9458e79686fd5c0742"}
Oct 29 22:13:43.561003 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.560919    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn9z2" event={"ID":"23921851-35c8-450c-a184-fe5ed0b1c6a7","Type":"ContainerStarted","Data":"044bc26e92f3f0b48c48ddfb35af3edabddcd9e052b53d32258b34a2bcf45694"}
Oct 29 22:13:43.575565 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.575521    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pvswk" event={"ID":"f3c107fc-072e-45ce-b4cd-acbe1284be88","Type":"ContainerStarted","Data":"812136f19d236521b0d7ad1f159dc08dcb7cccaa6850c0e91d3918dcd61cf519"}
Oct 29 22:13:43.589336 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.589295    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6" event={"ID":"198638bc-ff2e-4c81-a599-616e7b015749","Type":"ContainerStarted","Data":"10b41b6adef4be966c582cdc48e40fadb28dd646302d456467848faed34ec445"}
Oct 29 22:13:43.611880 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.611783    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v552v" event={"ID":"1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21","Type":"ContainerStarted","Data":"7b9a07980d4585ffee63d09c704ea6c68527a26286578c28d44f244aaf634d12"}
Oct 29 22:13:43.646221 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.646175    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal" event={"ID":"4504dba633558f5dac66e017907a0e24","Type":"ContainerStarted","Data":"e075d51ca14e4f7237ba867d6845ddcd67c963a12ee720c3c8786c47ebcebac9"}
Oct 29 22:13:43.781185 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:43.781150    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:13:44.043147 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:44.042254    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:44.043147 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:44.042318    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:44.043147 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:44.042471    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:44.043147 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:44.042546    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret podName:02990faa-cba3-4132-9ca7-fe0f7202bbe5 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:46.042516632 +0000 UTC m=+5.062790892 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret") pod "global-pull-secret-syncer-p5bdf" (UID: "02990faa-cba3-4132-9ca7-fe0f7202bbe5") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:44.043147 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:44.042982    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:44.043147 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:44.043033    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs podName:9b3ec29f-f4e7-4811-83ab-50680e0e60e7 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:46.04301673 +0000 UTC m=+5.063290991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs") pod "network-metrics-daemon-t48mn" (UID: "9b3ec29f-f4e7-4811-83ab-50680e0e60e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:44.143922 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:44.143158    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvr8w\" (UniqueName: \"kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w\") pod \"network-check-target-5pq4b\" (UID: \"c03b55ec-8ed6-42d6-895c-2e7213a2ba39\") " pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:13:44.143922 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:44.143337    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:44.143922 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:44.143362    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:44.143922 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:44.143377    2572 projected.go:196] Error preparing data for projected volume kube-api-access-bvr8w for pod openshift-network-diagnostics/network-check-target-5pq4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:44.143922 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:44.143439    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w podName:c03b55ec-8ed6-42d6-895c-2e7213a2ba39 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:46.143420459 +0000 UTC m=+5.163694726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bvr8w" (UniqueName: "kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w") pod "network-check-target-5pq4b" (UID: "c03b55ec-8ed6-42d6-895c-2e7213a2ba39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:44.371133 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:44.371021    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:08:42 +0000 UTC" deadline="2028-05-27 07:43:00.690365294 +0000 UTC"
Oct 29 22:13:44.371133 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:44.371065    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13809h29m16.319305122s"
Oct 29 22:13:44.445713 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:44.445649    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:44.445905 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:44.445799    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5bdf" podUID="02990faa-cba3-4132-9ca7-fe0f7202bbe5"
Oct 29 22:13:44.446183 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:44.446002    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:13:44.446183 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:44.446143    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pq4b" podUID="c03b55ec-8ed6-42d6-895c-2e7213a2ba39"
Oct 29 22:13:45.460129 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:45.458406    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:45.460129 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:45.458643    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t48mn" podUID="9b3ec29f-f4e7-4811-83ab-50680e0e60e7"
Oct 29 22:13:46.076619 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:46.076554    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:46.076619 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:46.076626    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:46.076837 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:46.076810    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:46.078003 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:46.076898    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret podName:02990faa-cba3-4132-9ca7-fe0f7202bbe5 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:50.076877716 +0000 UTC m=+9.097151976 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret") pod "global-pull-secret-syncer-p5bdf" (UID: "02990faa-cba3-4132-9ca7-fe0f7202bbe5") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:46.078003 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:46.077350    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:46.078003 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:46.077406    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs podName:9b3ec29f-f4e7-4811-83ab-50680e0e60e7 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:50.077389185 +0000 UTC m=+9.097663445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs") pod "network-metrics-daemon-t48mn" (UID: "9b3ec29f-f4e7-4811-83ab-50680e0e60e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:46.178090 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:46.178060    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvr8w\" (UniqueName: \"kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w\") pod \"network-check-target-5pq4b\" (UID: \"c03b55ec-8ed6-42d6-895c-2e7213a2ba39\") " pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:13:46.178285 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:46.178260    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:46.178285 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:46.178283    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:46.178437 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:46.178298    2572 projected.go:196] Error preparing data for projected volume kube-api-access-bvr8w for pod openshift-network-diagnostics/network-check-target-5pq4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:46.178437 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:46.178359    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w podName:c03b55ec-8ed6-42d6-895c-2e7213a2ba39 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:50.178337828 +0000 UTC m=+9.198612085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bvr8w" (UniqueName: "kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w") pod "network-check-target-5pq4b" (UID: "c03b55ec-8ed6-42d6-895c-2e7213a2ba39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:46.445189 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:46.445103    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:46.445394 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:46.445260    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5bdf" podUID="02990faa-cba3-4132-9ca7-fe0f7202bbe5"
Oct 29 22:13:46.445713 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:46.445520    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:13:46.445713 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:46.445654    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pq4b" podUID="c03b55ec-8ed6-42d6-895c-2e7213a2ba39"
Oct 29 22:13:47.449774 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:47.448953    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:47.449774 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:47.449372    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t48mn" podUID="9b3ec29f-f4e7-4811-83ab-50680e0e60e7"
Oct 29 22:13:47.672103 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:47.672058    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal" event={"ID":"c506ffe79e14398e87cdea51ca1d1f0f","Type":"ContainerStarted","Data":"5a15770ec5bac91693e0518d9cb121c031c8d3b3a9c3932c948f968bac048daf"}
Oct 29 22:13:47.675685 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:47.675647    2572 generic.go:356] "Generic (PLEG): container finished" podID="23921851-35c8-450c-a184-fe5ed0b1c6a7" containerID="cafd1faddf108ece9a07c7e7cc263b6b4f221f6bfd8515eea57a9543e437620c" exitCode=0
Oct 29 22:13:47.676589 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:47.675746    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn9z2" event={"ID":"23921851-35c8-450c-a184-fe5ed0b1c6a7","Type":"ContainerDied","Data":"cafd1faddf108ece9a07c7e7cc263b6b4f221f6bfd8515eea57a9543e437620c"}
Oct 29 22:13:47.680290 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:47.679916    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v552v" event={"ID":"1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21","Type":"ContainerStarted","Data":"af8ae3742f6d98e0fe31c4fc213a40c232193f85104d7b306959c21a8d027675"}
Oct 29 22:13:47.686704 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:47.686086    2572 generic.go:356] "Generic (PLEG): container finished" podID="4504dba633558f5dac66e017907a0e24" containerID="d3ff5a021aa4cf406ec0b427aaeb0a7addf126dd687ebb60f3e1d612a4098d93" exitCode=0
Oct 29 22:13:47.686704 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:47.686137    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal" event={"ID":"4504dba633558f5dac66e017907a0e24","Type":"ContainerDied","Data":"d3ff5a021aa4cf406ec0b427aaeb0a7addf126dd687ebb60f3e1d612a4098d93"}
Oct 29 22:13:47.713443 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:47.713136    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal" podStartSLOduration=5.713116293 podStartE2EDuration="5.713116293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:47.712500784 +0000 UTC m=+6.732775062" watchObservedRunningTime="2026-10-29 22:13:47.713116293 +0000 UTC m=+6.733390572"
Oct 29 22:13:47.756288 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:47.756097    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v552v" podStartSLOduration=2.040384113 podStartE2EDuration="6.756075909s" totalImagesPullingTime="4.715691796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:47.755042899 +0000 UTC m=+6.775317180" watchObservedRunningTime="2026-10-29 22:13:47.756075909 +0000 UTC m=+6.776350191"
Oct 29 22:13:48.445986 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:48.445236    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:48.445986 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:48.445388    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5bdf" podUID="02990faa-cba3-4132-9ca7-fe0f7202bbe5"
Oct 29 22:13:48.445986 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:48.445823    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:13:48.445986 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:48.445946    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pq4b" podUID="c03b55ec-8ed6-42d6-895c-2e7213a2ba39"
Oct 29 22:13:48.694684 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:48.694630    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal" event={"ID":"4504dba633558f5dac66e017907a0e24","Type":"ContainerStarted","Data":"0d9ce6c8ecd4c9cd0cfd9fb5192906b08ab1d0f0b16cf4351da47d9d8c78f9d3"}
Oct 29 22:13:48.733938 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:48.733873    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal" podStartSLOduration=6.733854219 podStartE2EDuration="6.733854219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:48.733170126 +0000 UTC m=+7.753444428" watchObservedRunningTime="2026-10-29 22:13:48.733854219 +0000 UTC m=+7.754128518"
Oct 29 22:13:49.447294 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:49.447263    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:49.447450 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:49.447381    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t48mn" podUID="9b3ec29f-f4e7-4811-83ab-50680e0e60e7"
Oct 29 22:13:49.698434 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:49.698393    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8bl7p" event={"ID":"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55","Type":"ContainerStarted","Data":"a3417b959fa2ebc43e04f9d9a979cdedd31905bf0d8d09a0340379bf6b3202d0"}
Oct 29 22:13:49.702775 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:49.702744    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw" event={"ID":"486bbe97-6300-49e9-a0f3-c9d316333048","Type":"ContainerStarted","Data":"9752031bae463cdf80c0440e6e5e5cf27cef4931bbc1891001f72ca960035cdd"}
Oct 29 22:13:49.743050 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:49.742989    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8bl7p" podStartSLOduration=2.9536667960000003 podStartE2EDuration="8.742968404s" totalImagesPullingTime="5.789301608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:49.742121133 +0000 UTC m=+8.762395408" watchObservedRunningTime="2026-10-29 22:13:49.742968404 +0000 UTC m=+8.763242679"
Oct 29 22:13:50.112716 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:50.112671    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:50.112934 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:50.112735    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:50.112934 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:50.112912    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:50.113037 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:50.112972    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret podName:02990faa-cba3-4132-9ca7-fe0f7202bbe5 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:58.112955172 +0000 UTC m=+17.133229430 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret") pod "global-pull-secret-syncer-p5bdf" (UID: "02990faa-cba3-4132-9ca7-fe0f7202bbe5") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:50.113373 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:50.113353    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:50.113501 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:50.113405    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs podName:9b3ec29f-f4e7-4811-83ab-50680e0e60e7 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:58.113390949 +0000 UTC m=+17.133665204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs") pod "network-metrics-daemon-t48mn" (UID: "9b3ec29f-f4e7-4811-83ab-50680e0e60e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:50.213494 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:50.213428    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvr8w\" (UniqueName: \"kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w\") pod \"network-check-target-5pq4b\" (UID: \"c03b55ec-8ed6-42d6-895c-2e7213a2ba39\") " pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:13:50.213704 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:50.213683    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:50.213781 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:50.213709    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:50.213781 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:50.213724    2572 projected.go:196] Error preparing data for projected volume kube-api-access-bvr8w for pod openshift-network-diagnostics/network-check-target-5pq4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:50.213917 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:50.213785    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w podName:c03b55ec-8ed6-42d6-895c-2e7213a2ba39 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:58.213766923 +0000 UTC m=+17.234041178 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bvr8w" (UniqueName: "kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w") pod "network-check-target-5pq4b" (UID: "c03b55ec-8ed6-42d6-895c-2e7213a2ba39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:50.445330 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:50.445294    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:13:50.445495 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:50.445434    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pq4b" podUID="c03b55ec-8ed6-42d6-895c-2e7213a2ba39"
Oct 29 22:13:50.445849 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:50.445831    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:50.445958 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:50.445928    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5bdf" podUID="02990faa-cba3-4132-9ca7-fe0f7202bbe5"
Oct 29 22:13:50.923768 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:50.923737    2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 22:13:51.397092 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:51.396892    2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:13:50.923761563Z","UUID":"bbc71ec1-1411-403b-b11b-73db5287ffd7","Handler":null,"Name":"","Endpoint":""}
Oct 29 22:13:51.400276 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:51.400248    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 22:13:51.400418 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:51.400291    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 22:13:51.446318 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:51.446290    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:51.446480 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:51.446404    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t48mn" podUID="9b3ec29f-f4e7-4811-83ab-50680e0e60e7"
Oct 29 22:13:51.711393 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:51.711357    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7q7dr" event={"ID":"37a98266-fad3-440d-95d4-85de9ad6f476","Type":"ContainerStarted","Data":"4ffe403020125e0fa9b6b9f3c637a91b756d0645254a8a56f663f0cf8b006797"}
Oct 29 22:13:51.713653 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:51.713624    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw" event={"ID":"486bbe97-6300-49e9-a0f3-c9d316333048","Type":"ContainerStarted","Data":"5b0a38a372196f08acb924ad43aa77a55e1cc338ec86ace510e960515d6769e5"}
Oct 29 22:13:52.008729 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:52.008659    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v552v"
Oct 29 22:13:52.010161 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:52.009957    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v552v"
Oct 29 22:13:52.047821 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:52.047750    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7q7dr" podStartSLOduration=2.41330106 podStartE2EDuration="11.047735519s" totalImagesPullingTime="8.634434459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:51.753452999 +0000 UTC m=+10.773727274" watchObservedRunningTime="2026-10-29 22:13:52.047735519 +0000 UTC m=+11.068009792"
Oct 29 22:13:52.445600 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:52.445559    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:13:52.445755 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:52.445699    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pq4b" podUID="c03b55ec-8ed6-42d6-895c-2e7213a2ba39"
Oct 29 22:13:52.446133 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:52.446110    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:52.446284 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:52.446251    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5bdf" podUID="02990faa-cba3-4132-9ca7-fe0f7202bbe5"
Oct 29 22:13:52.718053 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:52.718008    2572 generic.go:356] "Generic (PLEG): container finished" podID="23921851-35c8-450c-a184-fe5ed0b1c6a7" containerID="8adcd58764927d9b5d90059d1fa7dcf695570f568f363f21120a96d88938d8c7" exitCode=0
Oct 29 22:13:52.718240 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:52.718122    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn9z2" event={"ID":"23921851-35c8-450c-a184-fe5ed0b1c6a7","Type":"ContainerDied","Data":"8adcd58764927d9b5d90059d1fa7dcf695570f568f363f21120a96d88938d8c7"}
Oct 29 22:13:52.721527 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:52.721496    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jls5s" event={"ID":"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de","Type":"ContainerStarted","Data":"e9adda5d4f43511fedc1e405302f53d85c481233576b01227c3f3c801f42d218"}
Oct 29 22:13:52.724028 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:52.723985    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw" event={"ID":"486bbe97-6300-49e9-a0f3-c9d316333048","Type":"ContainerStarted","Data":"3a5cce34ff2a35db483901a5eaa2d9679f67780899bac10a4e5dac1779c7eb6f"}
Oct 29 22:13:52.807715 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:52.807655    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw" podStartSLOduration=2.868460099 podStartE2EDuration="11.807636737s" totalImagesPullingTime="8.939176638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:52.807382266 +0000 UTC m=+11.827656533" watchObservedRunningTime="2026-10-29 22:13:52.807636737 +0000 UTC m=+11.827911014"
Oct 29 22:13:52.844327 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:52.844269    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-jls5s" podStartSLOduration=1.820153101 podStartE2EDuration="10.844251097s" totalImagesPullingTime="9.024097996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:52.844114545 +0000 UTC m=+11.864388832" watchObservedRunningTime="2026-10-29 22:13:52.844251097 +0000 UTC m=+11.864525371"
Oct 29 22:13:53.446648 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:53.446060    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:53.446648 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:53.446397    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t48mn" podUID="9b3ec29f-f4e7-4811-83ab-50680e0e60e7"
Oct 29 22:13:53.728323 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:53.728286    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xfm9m" event={"ID":"cc8f498e-4039-43ef-91f7-036179511f01","Type":"ContainerStarted","Data":"6f2d969837576ef4c65876ea51c0c79afc56b72a08c158e89a013aab5ecbdadc"}
Oct 29 22:13:53.730673 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:53.730646    2572 generic.go:356] "Generic (PLEG): container finished" podID="23921851-35c8-450c-a184-fe5ed0b1c6a7" containerID="8325e667285f3c571437d602d542e16d92455faec8a989b223974a61ae5c604c" exitCode=0
Oct 29 22:13:53.730833 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:53.730729    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn9z2" event={"ID":"23921851-35c8-450c-a184-fe5ed0b1c6a7","Type":"ContainerDied","Data":"8325e667285f3c571437d602d542e16d92455faec8a989b223974a61ae5c604c"}
Oct 29 22:13:53.770220 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:53.770153    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xfm9m" podStartSLOduration=3.035290807 podStartE2EDuration="11.770136803s" totalImagesPullingTime="8.734845996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:53.770106429 +0000 UTC m=+12.790380705" watchObservedRunningTime="2026-10-29 22:13:53.770136803 +0000 UTC m=+12.790411078"
Oct 29 22:13:54.445047 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:54.445018    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:13:54.445197 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:54.445148    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pq4b" podUID="c03b55ec-8ed6-42d6-895c-2e7213a2ba39"
Oct 29 22:13:54.445268 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:54.445199    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:54.445326 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:54.445305    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5bdf" podUID="02990faa-cba3-4132-9ca7-fe0f7202bbe5"
Oct 29 22:13:54.736363 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:54.736317    2572 generic.go:356] "Generic (PLEG): container finished" podID="23921851-35c8-450c-a184-fe5ed0b1c6a7" containerID="c478d1186d378edf7897599cca23cc28e9e3c65f67e35ff317683462457c52b3" exitCode=0
Oct 29 22:13:54.736945 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:54.736426    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn9z2" event={"ID":"23921851-35c8-450c-a184-fe5ed0b1c6a7","Type":"ContainerDied","Data":"c478d1186d378edf7897599cca23cc28e9e3c65f67e35ff317683462457c52b3"}
Oct 29 22:13:54.739038 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:54.739011    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pvswk" event={"ID":"f3c107fc-072e-45ce-b4cd-acbe1284be88","Type":"ContainerStarted","Data":"c15395653946cba046506268abf0654d1dbc1ca7afde7cd4971a95333dc80055"}
Oct 29 22:13:54.842961 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:54.842912    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pvswk" podStartSLOduration=1.914977808 podStartE2EDuration="12.842891625s" totalImagesPullingTime="10.927913817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:54.842463627 +0000 UTC m=+13.862737902" watchObservedRunningTime="2026-10-29 22:13:54.842891625 +0000 UTC m=+13.863165900"
Oct 29 22:13:55.446044 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:55.446003    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:55.446206 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:55.446134    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t48mn" podUID="9b3ec29f-f4e7-4811-83ab-50680e0e60e7"
Oct 29 22:13:56.445283 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:56.445248    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:56.445934 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:56.445257    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:13:56.445934 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:56.445387    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5bdf" podUID="02990faa-cba3-4132-9ca7-fe0f7202bbe5"
Oct 29 22:13:56.445934 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:56.445474    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pq4b" podUID="c03b55ec-8ed6-42d6-895c-2e7213a2ba39"
Oct 29 22:13:56.746428 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:56.746388    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn9z2" event={"ID":"23921851-35c8-450c-a184-fe5ed0b1c6a7","Type":"ContainerStarted","Data":"cab18108f05777d958c4d1fd9caa34044465677ca33f285a2a6404ebc0033933"}
Oct 29 22:13:57.444527 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:57.444500    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:57.444679 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:57.444606    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t48mn" podUID="9b3ec29f-f4e7-4811-83ab-50680e0e60e7"
Oct 29 22:13:57.750093 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:57.750018    2572 generic.go:356] "Generic (PLEG): container finished" podID="23921851-35c8-450c-a184-fe5ed0b1c6a7" containerID="cab18108f05777d958c4d1fd9caa34044465677ca33f285a2a6404ebc0033933" exitCode=0
Oct 29 22:13:57.750093 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:57.750064    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn9z2" event={"ID":"23921851-35c8-450c-a184-fe5ed0b1c6a7","Type":"ContainerDied","Data":"cab18108f05777d958c4d1fd9caa34044465677ca33f285a2a6404ebc0033933"}
Oct 29 22:13:58.183851 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:58.183662    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:58.183969 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:58.183905    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:58.183969 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:58.183801    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:58.184034 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:58.183998    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret podName:02990faa-cba3-4132-9ca7-fe0f7202bbe5 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:14.183976957 +0000 UTC m=+33.204251210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret") pod "global-pull-secret-syncer-p5bdf" (UID: "02990faa-cba3-4132-9ca7-fe0f7202bbe5") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:58.184034 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:58.184002    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:58.184034 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:58.184033    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs podName:9b3ec29f-f4e7-4811-83ab-50680e0e60e7 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:14.184024153 +0000 UTC m=+33.204298419 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs") pod "network-metrics-daemon-t48mn" (UID: "9b3ec29f-f4e7-4811-83ab-50680e0e60e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:58.284708 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:58.284680    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvr8w\" (UniqueName: \"kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w\") pod \"network-check-target-5pq4b\" (UID: \"c03b55ec-8ed6-42d6-895c-2e7213a2ba39\") " pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:13:58.284861 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:58.284790    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:58.284861 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:58.284828    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:58.284861 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:58.284840    2572 projected.go:196] Error preparing data for projected volume kube-api-access-bvr8w for pod openshift-network-diagnostics/network-check-target-5pq4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:58.285019 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:58.284885    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w podName:c03b55ec-8ed6-42d6-895c-2e7213a2ba39 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:14.284873103 +0000 UTC m=+33.305147369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bvr8w" (UniqueName: "kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w") pod "network-check-target-5pq4b" (UID: "c03b55ec-8ed6-42d6-895c-2e7213a2ba39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:58.444547 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:58.444510    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:13:58.444727 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:58.444625    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pq4b" podUID="c03b55ec-8ed6-42d6-895c-2e7213a2ba39"
Oct 29 22:13:58.444727 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:58.444657    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:13:58.444727 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:58.444710    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5bdf" podUID="02990faa-cba3-4132-9ca7-fe0f7202bbe5"
Oct 29 22:13:58.755888 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:58.755856    2572 generic.go:356] "Generic (PLEG): container finished" podID="23921851-35c8-450c-a184-fe5ed0b1c6a7" containerID="67e7f9ec686032ba4ec1745c0ceeb139e6b123dd6d97d256b9853f882083ccb4" exitCode=0
Oct 29 22:13:58.756384 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:58.755957    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn9z2" event={"ID":"23921851-35c8-450c-a184-fe5ed0b1c6a7","Type":"ContainerDied","Data":"67e7f9ec686032ba4ec1745c0ceeb139e6b123dd6d97d256b9853f882083ccb4"}
Oct 29 22:13:58.761326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:58.761308    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:13:58.761719 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:58.761680    2572 generic.go:356] "Generic (PLEG): container finished" podID="198638bc-ff2e-4c81-a599-616e7b015749" containerID="ce9ec1ef6833ca17735f0151ac7ac8762b21a8bbe3505e4fcae77900fa421af2" exitCode=1
Oct 29 22:13:58.761830 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:58.761731    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6" event={"ID":"198638bc-ff2e-4c81-a599-616e7b015749","Type":"ContainerStarted","Data":"d96c527952e854031e9a37eb6d3aee033e9cbeb3825233c641ce7e8706cea88e"}
Oct 29 22:13:58.761830 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:58.761789    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6" event={"ID":"198638bc-ff2e-4c81-a599-616e7b015749","Type":"ContainerStarted","Data":"4d3bb3cd0e3d593d3e6d6742c4d50c12ae8f785ec5050e5dacdc62db634a8075"}
Oct 29 22:13:58.761923 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:58.761842    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6" event={"ID":"198638bc-ff2e-4c81-a599-616e7b015749","Type":"ContainerStarted","Data":"77e6686842e8f56fcf29cf6052f355f17f8336251d70954b1a72abf98551becb"}
Oct 29 22:13:58.761923 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:58.761871    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6" event={"ID":"198638bc-ff2e-4c81-a599-616e7b015749","Type":"ContainerStarted","Data":"31d21398719c5e38654d79284704cb21e4d94335800cd612176b20323ac3432f"}
Oct 29 22:13:58.761923 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:58.761897    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6" event={"ID":"198638bc-ff2e-4c81-a599-616e7b015749","Type":"ContainerDied","Data":"ce9ec1ef6833ca17735f0151ac7ac8762b21a8bbe3505e4fcae77900fa421af2"}
Oct 29 22:13:58.762070 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:58.761925    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6" event={"ID":"198638bc-ff2e-4c81-a599-616e7b015749","Type":"ContainerStarted","Data":"6ca638176df1220e8cc96b24c5299e15e412b5380085d65d1be038fef6ec3089"}
Oct 29 22:13:59.445233 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:59.445046    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:13:59.445374 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:13:59.445319    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t48mn" podUID="9b3ec29f-f4e7-4811-83ab-50680e0e60e7"
Oct 29 22:13:59.767127 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:59.767046    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn9z2" event={"ID":"23921851-35c8-450c-a184-fe5ed0b1c6a7","Type":"ContainerStarted","Data":"b8ce70d9904a11c33cc4bb43ce1dfddb89d0af143bd1f1873dce0ea4ce27cfaf"}
Oct 29 22:13:59.835644 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:13:59.835596    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bn9z2" podStartSLOduration=7.100594765 podStartE2EDuration="17.835583153s" totalImagesPullingTime="10.734988388s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:59.835390094 +0000 UTC m=+18.855664370" watchObservedRunningTime="2026-10-29 22:13:59.835583153 +0000 UTC m=+18.855857427"
Oct 29 22:14:00.445501 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:00.445458    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:14:00.445501 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:00.445493    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:00.445703 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:00.445569    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pq4b" podUID="c03b55ec-8ed6-42d6-895c-2e7213a2ba39"
Oct 29 22:14:00.445703 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:00.445677    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5bdf" podUID="02990faa-cba3-4132-9ca7-fe0f7202bbe5"
Oct 29 22:14:00.772620 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:00.772533    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:14:00.773018 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:00.772889    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6" event={"ID":"198638bc-ff2e-4c81-a599-616e7b015749","Type":"ContainerStarted","Data":"7605db4495294f1ea645d508b05fcef4e10a8a5a42547348a8b39ffa429cceca"}
Oct 29 22:14:01.315793 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:01.315764    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v552v"
Oct 29 22:14:01.316000 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:01.315900    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:14:01.316352 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:01.316337    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v552v"
Oct 29 22:14:01.445164 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:01.445135    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:14:01.445291 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:01.445224    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t48mn" podUID="9b3ec29f-f4e7-4811-83ab-50680e0e60e7"
Oct 29 22:14:02.445250 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:02.445202    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:14:02.445782 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:02.445202    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:02.445782 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:02.445320    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pq4b" podUID="c03b55ec-8ed6-42d6-895c-2e7213a2ba39"
Oct 29 22:14:02.445782 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:02.445459    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5bdf" podUID="02990faa-cba3-4132-9ca7-fe0f7202bbe5"
Oct 29 22:14:02.780648 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:02.780571    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:14:02.780953 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:02.780933    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6" event={"ID":"198638bc-ff2e-4c81-a599-616e7b015749","Type":"ContainerStarted","Data":"fcd343933a4725017332f29874c8cd45e4c2d20b3be7d753817b71a313b5e5df"}
Oct 29 22:14:02.781303 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:02.781280    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:02.781422 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:02.781348    2572 scope.go:122] "RemoveContainer" containerID="ce9ec1ef6833ca17735f0151ac7ac8762b21a8bbe3505e4fcae77900fa421af2"
Oct 29 22:14:02.781422 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:02.781356    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:02.799019 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:02.798991    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:03.444717 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:03.444530    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:14:03.445041 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:03.444801    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t48mn" podUID="9b3ec29f-f4e7-4811-83ab-50680e0e60e7"
Oct 29 22:14:03.792029 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:03.791956    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:14:03.792898 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:03.792364    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6" event={"ID":"198638bc-ff2e-4c81-a599-616e7b015749","Type":"ContainerStarted","Data":"7a0edde4936b23dd4ff66d45ce1cef4e8818f43170fef79b2b58bd74801cf4c4"}
Oct 29 22:14:03.792898 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:03.792596    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:03.817012 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:03.816302    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:03.876860 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:03.876786    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6" podStartSLOduration=7.537479004 podStartE2EDuration="22.876773422s" totalImagesPullingTime="15.339294418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:03.876469609 +0000 UTC m=+22.896743900" watchObservedRunningTime="2026-10-29 22:14:03.876773422 +0000 UTC m=+22.897047738"
Oct 29 22:14:04.444517 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:04.444483    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:14:04.444696 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:04.444622    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pq4b" podUID="c03b55ec-8ed6-42d6-895c-2e7213a2ba39"
Oct 29 22:14:04.444696 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:04.444671    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:04.444904 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:04.444777    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5bdf" podUID="02990faa-cba3-4132-9ca7-fe0f7202bbe5"
Oct 29 22:14:04.757678 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:04.757601    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p5bdf"]
Oct 29 22:14:04.760960 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:04.760929    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t48mn"]
Oct 29 22:14:04.761111 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:04.761075    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:14:04.761208 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:04.761188    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t48mn" podUID="9b3ec29f-f4e7-4811-83ab-50680e0e60e7"
Oct 29 22:14:04.761833 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:04.761797    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5pq4b"]
Oct 29 22:14:04.794521 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:04.794493    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:14:04.794933 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:04.794493    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:04.794933 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:04.794586    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pq4b" podUID="c03b55ec-8ed6-42d6-895c-2e7213a2ba39"
Oct 29 22:14:04.794933 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:04.794685    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5bdf" podUID="02990faa-cba3-4132-9ca7-fe0f7202bbe5"
Oct 29 22:14:06.445289 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:06.445250    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:14:06.445289 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:06.445281    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:06.445771 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:06.445352    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pq4b" podUID="c03b55ec-8ed6-42d6-895c-2e7213a2ba39"
Oct 29 22:14:06.445771 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:06.445380    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:14:06.445771 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:06.445456    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t48mn" podUID="9b3ec29f-f4e7-4811-83ab-50680e0e60e7"
Oct 29 22:14:06.445771 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:06.445519    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5bdf" podUID="02990faa-cba3-4132-9ca7-fe0f7202bbe5"
Oct 29 22:14:08.444740 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:08.444700    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:14:08.444740 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:08.444737    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:14:08.445167 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:08.444701    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:08.445167 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:08.444842    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t48mn" podUID="9b3ec29f-f4e7-4811-83ab-50680e0e60e7"
Oct 29 22:14:08.445167 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:08.444952    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5bdf" podUID="02990faa-cba3-4132-9ca7-fe0f7202bbe5"
Oct 29 22:14:08.445167 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:08.445014    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pq4b" podUID="c03b55ec-8ed6-42d6-895c-2e7213a2ba39"
Oct 29 22:14:10.338139 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.338113    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeReady"
Oct 29 22:14:10.338535 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.338234    2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 29 22:14:10.389432 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.389401    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xp49w"]
Oct 29 22:14:10.392758 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.392738    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:10.392891 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.392743    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fzpgj"]
Oct 29 22:14:10.395659 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.395637    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fzpgj"
Oct 29 22:14:10.411411 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.411385    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 22:14:10.411411 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.411407    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2nqlz\""
Oct 29 22:14:10.411564 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.411393    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 22:14:10.412721 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.412703    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 22:14:10.413996 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.413975    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xp49w"]
Oct 29 22:14:10.417086 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.417066    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fzpgj"]
Oct 29 22:14:10.444640 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.444603    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:10.444804 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.444646    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:14:10.444804 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.444692    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:14:10.516921 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.516891    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xsn6t\""
Oct 29 22:14:10.516921 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.516896    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 22:14:10.517168 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.517009    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-977wj\""
Oct 29 22:14:10.517168 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.517062    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 22:14:10.517168 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.517073    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 22:14:10.517168 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.517068    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7vbxc\""
Oct 29 22:14:10.517168 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.517076    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 22:14:10.517168 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.517108    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 22:14:10.517168 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.517173    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 22:14:10.517379 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.517236    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-v7npq\""
Oct 29 22:14:10.567622 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.567587    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5blnb\" (UniqueName: \"kubernetes.io/projected/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-kube-api-access-5blnb\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:10.567740 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.567655    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-config-volume\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:10.567740 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.567711    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d637bbed-2662-4313-9639-e7a3bc148955-cert\") pod \"ingress-canary-fzpgj\" (UID: \"d637bbed-2662-4313-9639-e7a3bc148955\") " pod="openshift-ingress-canary/ingress-canary-fzpgj"
Oct 29 22:14:10.567889 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.567838    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm7gg\" (UniqueName: \"kubernetes.io/projected/d637bbed-2662-4313-9639-e7a3bc148955-kube-api-access-zm7gg\") pod \"ingress-canary-fzpgj\" (UID: \"d637bbed-2662-4313-9639-e7a3bc148955\") " pod="openshift-ingress-canary/ingress-canary-fzpgj"
Oct 29 22:14:10.567944 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.567907    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-tmp-dir\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:10.567944 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.567938    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:10.668964 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.668922    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-tmp-dir\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:10.668964 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.668969    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:10.669143 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.669116    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5blnb\" (UniqueName: \"kubernetes.io/projected/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-kube-api-access-5blnb\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:10.669143 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:10.669138    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:10.669229 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:10.669198    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls podName:d4a55136-e7f7-46ac-b58e-24e95ab0aea8 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:11.169179228 +0000 UTC m=+30.189453500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls") pod "dns-default-xp49w" (UID: "d4a55136-e7f7-46ac-b58e-24e95ab0aea8") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:10.669229 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.669219    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-config-volume\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:10.669326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.669242    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d637bbed-2662-4313-9639-e7a3bc148955-cert\") pod \"ingress-canary-fzpgj\" (UID: \"d637bbed-2662-4313-9639-e7a3bc148955\") " pod="openshift-ingress-canary/ingress-canary-fzpgj"
Oct 29 22:14:10.669326 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.669281    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7gg\" (UniqueName: \"kubernetes.io/projected/d637bbed-2662-4313-9639-e7a3bc148955-kube-api-access-zm7gg\") pod \"ingress-canary-fzpgj\" (UID: \"d637bbed-2662-4313-9639-e7a3bc148955\") " pod="openshift-ingress-canary/ingress-canary-fzpgj"
Oct 29 22:14:10.669414 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.669350    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-tmp-dir\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:10.669793 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.669770    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-config-volume\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:10.673532 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.673413    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d637bbed-2662-4313-9639-e7a3bc148955-cert\") pod \"ingress-canary-fzpgj\" (UID: \"d637bbed-2662-4313-9639-e7a3bc148955\") " pod="openshift-ingress-canary/ingress-canary-fzpgj"
Oct 29 22:14:10.689136 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.689110    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7gg\" (UniqueName: \"kubernetes.io/projected/d637bbed-2662-4313-9639-e7a3bc148955-kube-api-access-zm7gg\") pod \"ingress-canary-fzpgj\" (UID: \"d637bbed-2662-4313-9639-e7a3bc148955\") " pod="openshift-ingress-canary/ingress-canary-fzpgj"
Oct 29 22:14:10.689136 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.689126    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5blnb\" (UniqueName: \"kubernetes.io/projected/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-kube-api-access-5blnb\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:10.709375 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:10.709342    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fzpgj"
Oct 29 22:14:11.016382 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:11.016305    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fzpgj"]
Oct 29 22:14:11.020743 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:14:11.020712    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd637bbed_2662_4313_9639_e7a3bc148955.slice/crio-c5c5bbf24408fd6a752a915c8039399c0009aee67effacb2986761558aef87bf WatchSource:0}: Error finding container c5c5bbf24408fd6a752a915c8039399c0009aee67effacb2986761558aef87bf: Status 404 returned error can't find the container with id c5c5bbf24408fd6a752a915c8039399c0009aee67effacb2986761558aef87bf
Oct 29 22:14:11.173443 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:11.173395    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:11.173640 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:11.173533    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:11.173640 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:11.173601    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls podName:d4a55136-e7f7-46ac-b58e-24e95ab0aea8 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:12.173584602 +0000 UTC m=+31.193858865 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls") pod "dns-default-xp49w" (UID: "d4a55136-e7f7-46ac-b58e-24e95ab0aea8") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:11.813006 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:11.812964    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fzpgj" event={"ID":"d637bbed-2662-4313-9639-e7a3bc148955","Type":"ContainerStarted","Data":"c5c5bbf24408fd6a752a915c8039399c0009aee67effacb2986761558aef87bf"}
Oct 29 22:14:12.182268 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:12.182232    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:12.182446 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:12.182405    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:12.182502 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:12.182480    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls podName:d4a55136-e7f7-46ac-b58e-24e95ab0aea8 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:14.182460954 +0000 UTC m=+33.202735220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls") pod "dns-default-xp49w" (UID: "d4a55136-e7f7-46ac-b58e-24e95ab0aea8") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:14.192757 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.192723    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dnx6b"]
Oct 29 22:14:14.196176 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.196148    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:14.196297 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.196190    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:14:14.196297 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.196213    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:14.197232 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:14.196532    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:14.197232 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:14.196685    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 29 22:14:14.197232 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:14.196766    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls podName:d4a55136-e7f7-46ac-b58e-24e95ab0aea8 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:18.19674022 +0000 UTC m=+37.217014473 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls") pod "dns-default-xp49w" (UID: "d4a55136-e7f7-46ac-b58e-24e95ab0aea8") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:14.197838 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:14.197482    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs podName:9b3ec29f-f4e7-4811-83ab-50680e0e60e7 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:46.197422502 +0000 UTC m=+65.217696780 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs") pod "network-metrics-daemon-t48mn" (UID: "9b3ec29f-f4e7-4811-83ab-50680e0e60e7") : secret "metrics-daemon-secret" not found
Oct 29 22:14:14.200564 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.200533    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:14.220121 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.220092    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:14.222565 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.222533    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dnx6b"]
Oct 29 22:14:14.256005 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.255971    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-hc4lm-rqsvk"]
Oct 29 22:14:14.268851 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.268800    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:14.278765 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.278740    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hc4lm-rqsvk"]
Oct 29 22:14:14.296649 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.296621    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvr8w\" (UniqueName: \"kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w\") pod \"network-check-target-5pq4b\" (UID: \"c03b55ec-8ed6-42d6-895c-2e7213a2ba39\") " pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:14:14.299177 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.299155    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvr8w\" (UniqueName: \"kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w\") pod \"network-check-target-5pq4b\" (UID: \"c03b55ec-8ed6-42d6-895c-2e7213a2ba39\") " pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:14:14.326280 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.326249    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 22:14:14.326428 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.326310    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 29 22:14:14.326539 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.326524    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 22:14:14.326608 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.326596    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 22:14:14.326638 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.326602    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 22:14:14.326727 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.326667    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xpwxr\""
Oct 29 22:14:14.327027 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.327013    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hkdgm\""
Oct 29 22:14:14.352447 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.352419    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 29 22:14:14.357269 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.357249    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:14.367558 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.367533    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:14:14.397372 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.397333    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f18ecc2-6ee2-47fa-befa-83065346a118-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:14.397466 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.397385    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qclfj\" (UniqueName: \"kubernetes.io/projected/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-api-access-qclfj\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:14.397511 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.397496    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-trusted-ca-bundle\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:14.397550 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.397518    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f18ecc2-6ee2-47fa-befa-83065346a118-crio-socket\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:14.397550 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.397545    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-service-ca-bundle\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:14.397622 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.397570    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f18ecc2-6ee2-47fa-befa-83065346a118-data-volume\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:14.397740 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.397713    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:14.397948 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.397775    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/807ed4de-272f-4bb2-bf26-cc33f6967771-archives-path\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:14.397948 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.397839    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq8zc\" (UniqueName: \"kubernetes.io/projected/807ed4de-272f-4bb2-bf26-cc33f6967771-kube-api-access-kq8zc\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:14.494901 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.494866    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5774f"]
Oct 29 22:14:14.498249 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.498224    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-service-ca-bundle\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:14.498383 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.498290    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f18ecc2-6ee2-47fa-befa-83065346a118-data-volume\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:14.498383 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.498324    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:14.498383 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.498349    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/807ed4de-272f-4bb2-bf26-cc33f6967771-archives-path\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:14.498547 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.498397    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq8zc\" (UniqueName: \"kubernetes.io/projected/807ed4de-272f-4bb2-bf26-cc33f6967771-kube-api-access-kq8zc\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:14.498547 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:14.498430    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 22:14:14.498547 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.498444    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f18ecc2-6ee2-47fa-befa-83065346a118-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:14.498547 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.498469    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qclfj\" (UniqueName: \"kubernetes.io/projected/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-api-access-qclfj\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:14.498547 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:14.498494    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-rbac-proxy-cm podName:8f18ecc2-6ee2-47fa-befa-83065346a118 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:14.998477086 +0000 UTC m=+34.018751362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnx6b" (UID: "8f18ecc2-6ee2-47fa-befa-83065346a118") : configmap "kube-rbac-proxy" not found
Oct 29 22:14:14.498786 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:14.498576    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:14.498786 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.498555    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-trusted-ca-bundle\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:14.498786 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.498624    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f18ecc2-6ee2-47fa-befa-83065346a118-crio-socket\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:14.498786 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:14.498643    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f18ecc2-6ee2-47fa-befa-83065346a118-insights-runtime-extractor-tls podName:8f18ecc2-6ee2-47fa-befa-83065346a118 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:14.998623943 +0000 UTC m=+34.018898211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f18ecc2-6ee2-47fa-befa-83065346a118-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnx6b" (UID: "8f18ecc2-6ee2-47fa-befa-83065346a118") : secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:14.498786 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.498737    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f18ecc2-6ee2-47fa-befa-83065346a118-data-volume\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:14.499060 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.498756    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/807ed4de-272f-4bb2-bf26-cc33f6967771-archives-path\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:14.499060 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.498835    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f18ecc2-6ee2-47fa-befa-83065346a118-crio-socket\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:14.499060 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.499022    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-service-ca-bundle\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:14.499852 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.499831    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-trusted-ca-bundle\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:14.521691 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.521659    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qclfj\" (UniqueName: \"kubernetes.io/projected/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-api-access-qclfj\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:14.523086 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.523057    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq8zc\" (UniqueName: \"kubernetes.io/projected/807ed4de-272f-4bb2-bf26-cc33f6967771-kube-api-access-kq8zc\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:14.526057 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.526039    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5774f"]
Oct 29 22:14:14.526141 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.526129    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5774f"
Oct 29 22:14:14.593058 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.593025    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:14.618936 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.618887    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 29 22:14:14.618936 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.618887    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:14.619124 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.619049    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kj4hx\""
Oct 29 22:14:14.668677 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.668652    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p5bdf"]
Oct 29 22:14:14.672593 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:14:14.672069    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02990faa_cba3_4132_9ca7_fe0f7202bbe5.slice/crio-8d7c43ba30f9c0725e2136f33a7795b1e1bb07bdd28c8c1a1ebbd0cf9bd9e646 WatchSource:0}: Error finding container 8d7c43ba30f9c0725e2136f33a7795b1e1bb07bdd28c8c1a1ebbd0cf9bd9e646: Status 404 returned error can't find the container with id 8d7c43ba30f9c0725e2136f33a7795b1e1bb07bdd28c8c1a1ebbd0cf9bd9e646
Oct 29 22:14:14.691512 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.691486    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5pq4b"]
Oct 29 22:14:14.694186 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:14:14.694120    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03b55ec_8ed6_42d6_895c_2e7213a2ba39.slice/crio-cca931e0b108a89e3426252e0979e7dd87f22129f2bb5578ef6903f5ea74521e WatchSource:0}: Error finding container cca931e0b108a89e3426252e0979e7dd87f22129f2bb5578ef6903f5ea74521e: Status 404 returned error can't find the container with id cca931e0b108a89e3426252e0979e7dd87f22129f2bb5578ef6903f5ea74521e
Oct 29 22:14:14.700030 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.700006    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbxms\" (UniqueName: \"kubernetes.io/projected/f5cf9217-fa53-45ec-b7fa-a80520684473-kube-api-access-gbxms\") pod \"migrator-66df5d496b-5774f\" (UID: \"f5cf9217-fa53-45ec-b7fa-a80520684473\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5774f"
Oct 29 22:14:14.801374 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.801342    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxms\" (UniqueName: \"kubernetes.io/projected/f5cf9217-fa53-45ec-b7fa-a80520684473-kube-api-access-gbxms\") pod \"migrator-66df5d496b-5774f\" (UID: \"f5cf9217-fa53-45ec-b7fa-a80520684473\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5774f"
Oct 29 22:14:14.820985 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.820954    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxms\" (UniqueName: \"kubernetes.io/projected/f5cf9217-fa53-45ec-b7fa-a80520684473-kube-api-access-gbxms\") pod \"migrator-66df5d496b-5774f\" (UID: \"f5cf9217-fa53-45ec-b7fa-a80520684473\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5774f"
Oct 29 22:14:14.822003 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.821970    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fzpgj" event={"ID":"d637bbed-2662-4313-9639-e7a3bc148955","Type":"ContainerStarted","Data":"2bf81a1f270f8ac6caa47744ac8627cd6997bca19f48f4a206e57432fcad4669"}
Oct 29 22:14:14.823155 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.823128    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5pq4b" event={"ID":"c03b55ec-8ed6-42d6-895c-2e7213a2ba39","Type":"ContainerStarted","Data":"cca931e0b108a89e3426252e0979e7dd87f22129f2bb5578ef6903f5ea74521e"}
Oct 29 22:14:14.824177 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.824158    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p5bdf" event={"ID":"02990faa-cba3-4132-9ca7-fe0f7202bbe5","Type":"ContainerStarted","Data":"8d7c43ba30f9c0725e2136f33a7795b1e1bb07bdd28c8c1a1ebbd0cf9bd9e646"}
Oct 29 22:14:14.839911 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.839889    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5774f"
Oct 29 22:14:14.863227 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.863182    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fzpgj" podStartSLOduration=2.042550148 podStartE2EDuration="4.863167263s" totalImagesPullingTime="2.820617115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:14.862328549 +0000 UTC m=+33.882602834" watchObservedRunningTime="2026-10-29 22:14:14.863167263 +0000 UTC m=+33.883441537"
Oct 29 22:14:14.911432 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:14.911346    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hc4lm-rqsvk"]
Oct 29 22:14:15.002899 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.002788    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:15.002899 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.002879    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f18ecc2-6ee2-47fa-befa-83065346a118-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:15.003111 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:15.002981    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:15.003111 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:15.002995    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 22:14:15.003111 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:15.003040    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f18ecc2-6ee2-47fa-befa-83065346a118-insights-runtime-extractor-tls podName:8f18ecc2-6ee2-47fa-befa-83065346a118 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:16.003027931 +0000 UTC m=+35.023302207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f18ecc2-6ee2-47fa-befa-83065346a118-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnx6b" (UID: "8f18ecc2-6ee2-47fa-befa-83065346a118") : secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:15.003111 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:15.003055    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-rbac-proxy-cm podName:8f18ecc2-6ee2-47fa-befa-83065346a118 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:16.003048881 +0000 UTC m=+35.023323133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnx6b" (UID: "8f18ecc2-6ee2-47fa-befa-83065346a118") : configmap "kube-rbac-proxy" not found
Oct 29 22:14:15.169376 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.169301    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5774f"]
Oct 29 22:14:15.172342 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:14:15.172312    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cf9217_fa53_45ec_b7fa_a80520684473.slice/crio-de3a0df9870dc7439e231f189c64c887a86943e666eeae14f03380d0301dbf89 WatchSource:0}: Error finding container de3a0df9870dc7439e231f189c64c887a86943e666eeae14f03380d0301dbf89: Status 404 returned error can't find the container with id de3a0df9870dc7439e231f189c64c887a86943e666eeae14f03380d0301dbf89
Oct 29 22:14:15.482015 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.481983    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-x4nbg"]
Oct 29 22:14:15.505868 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.505836    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg"
Oct 29 22:14:15.506283 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.506252    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-x4nbg"]
Oct 29 22:14:15.521115 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.521071    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 29 22:14:15.521246 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.521118    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 29 22:14:15.521246 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.521141    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hl5hv\""
Oct 29 22:14:15.521246 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.521194    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 29 22:14:15.521246 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.521221    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 29 22:14:15.608480 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.608438    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3104d9ac-59f1-406b-9d63-7af7892b40bf-signing-cabundle\") pod \"service-ca-5fffc97bdf-x4nbg\" (UID: \"3104d9ac-59f1-406b-9d63-7af7892b40bf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg"
Oct 29 22:14:15.608647 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.608524    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qv6v\" (UniqueName: \"kubernetes.io/projected/3104d9ac-59f1-406b-9d63-7af7892b40bf-kube-api-access-5qv6v\") pod \"service-ca-5fffc97bdf-x4nbg\" (UID: \"3104d9ac-59f1-406b-9d63-7af7892b40bf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg"
Oct 29 22:14:15.608647 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.608562    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3104d9ac-59f1-406b-9d63-7af7892b40bf-signing-key\") pod \"service-ca-5fffc97bdf-x4nbg\" (UID: \"3104d9ac-59f1-406b-9d63-7af7892b40bf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg"
Oct 29 22:14:15.709606 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.709567    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3104d9ac-59f1-406b-9d63-7af7892b40bf-signing-cabundle\") pod \"service-ca-5fffc97bdf-x4nbg\" (UID: \"3104d9ac-59f1-406b-9d63-7af7892b40bf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg"
Oct 29 22:14:15.709770 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.709647    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qv6v\" (UniqueName: \"kubernetes.io/projected/3104d9ac-59f1-406b-9d63-7af7892b40bf-kube-api-access-5qv6v\") pod \"service-ca-5fffc97bdf-x4nbg\" (UID: \"3104d9ac-59f1-406b-9d63-7af7892b40bf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg"
Oct 29 22:14:15.709770 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.709683    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3104d9ac-59f1-406b-9d63-7af7892b40bf-signing-key\") pod \"service-ca-5fffc97bdf-x4nbg\" (UID: \"3104d9ac-59f1-406b-9d63-7af7892b40bf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg"
Oct 29 22:14:15.710934 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.710904    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3104d9ac-59f1-406b-9d63-7af7892b40bf-signing-cabundle\") pod \"service-ca-5fffc97bdf-x4nbg\" (UID: \"3104d9ac-59f1-406b-9d63-7af7892b40bf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg"
Oct 29 22:14:15.712909 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.712886    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3104d9ac-59f1-406b-9d63-7af7892b40bf-signing-key\") pod \"service-ca-5fffc97bdf-x4nbg\" (UID: \"3104d9ac-59f1-406b-9d63-7af7892b40bf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg"
Oct 29 22:14:15.731075 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.731038    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qv6v\" (UniqueName: \"kubernetes.io/projected/3104d9ac-59f1-406b-9d63-7af7892b40bf-kube-api-access-5qv6v\") pod \"service-ca-5fffc97bdf-x4nbg\" (UID: \"3104d9ac-59f1-406b-9d63-7af7892b40bf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg"
Oct 29 22:14:15.829196 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.829103    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5774f" event={"ID":"f5cf9217-fa53-45ec-b7fa-a80520684473","Type":"ContainerStarted","Data":"de3a0df9870dc7439e231f189c64c887a86943e666eeae14f03380d0301dbf89"}
Oct 29 22:14:15.829354 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.829239    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg"
Oct 29 22:14:15.830920 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:15.830891    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hc4lm-rqsvk" event={"ID":"807ed4de-272f-4bb2-bf26-cc33f6967771","Type":"ContainerStarted","Data":"b7ddd0d866bb33c277ffe06ba1ea76a766bc499124e08a77ebfdf1a4d28ba0d5"}
Oct 29 22:14:16.012825 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:16.012777    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f18ecc2-6ee2-47fa-befa-83065346a118-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:16.012994 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:16.012911    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:16.013068 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:16.013041    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 22:14:16.013133 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:16.013109    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-rbac-proxy-cm podName:8f18ecc2-6ee2-47fa-befa-83065346a118 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:18.013091887 +0000 UTC m=+37.033366145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnx6b" (UID: "8f18ecc2-6ee2-47fa-befa-83065346a118") : configmap "kube-rbac-proxy" not found
Oct 29 22:14:16.013206 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:16.013192    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:16.013261 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:16.013233    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f18ecc2-6ee2-47fa-befa-83065346a118-insights-runtime-extractor-tls podName:8f18ecc2-6ee2-47fa-befa-83065346a118 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:18.013220315 +0000 UTC m=+37.033494581 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f18ecc2-6ee2-47fa-befa-83065346a118-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnx6b" (UID: "8f18ecc2-6ee2-47fa-befa-83065346a118") : secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:16.163012 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:16.162977    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-x4nbg"]
Oct 29 22:14:16.170198 ip-10-0-138-2 kubenswrapper[2572]: W1029 22:14:16.170158    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3104d9ac_59f1_406b_9d63_7af7892b40bf.slice/crio-c4d57d45e00a8156835d17e56224cc17abe814c33632ca86409e4b9a6a9ddcfc WatchSource:0}: Error finding container c4d57d45e00a8156835d17e56224cc17abe814c33632ca86409e4b9a6a9ddcfc: Status 404 returned error can't find the container with id c4d57d45e00a8156835d17e56224cc17abe814c33632ca86409e4b9a6a9ddcfc
Oct 29 22:14:16.835924 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:16.835847    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg" event={"ID":"3104d9ac-59f1-406b-9d63-7af7892b40bf","Type":"ContainerStarted","Data":"c4d57d45e00a8156835d17e56224cc17abe814c33632ca86409e4b9a6a9ddcfc"}
Oct 29 22:14:17.841610 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:17.840844    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5774f" event={"ID":"f5cf9217-fa53-45ec-b7fa-a80520684473","Type":"ContainerStarted","Data":"eae2448794bb93fe05a7b4c178b30c7dd15f949b8e50431c2af20d663bdc1405"}
Oct 29 22:14:18.031147 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:18.031114    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f18ecc2-6ee2-47fa-befa-83065346a118-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:18.031318 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:18.031302    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:18.031397 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:18.031357    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:18.031397 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:18.031380    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f18ecc2-6ee2-47fa-befa-83065346a118-insights-runtime-extractor-tls podName:8f18ecc2-6ee2-47fa-befa-83065346a118 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:22.031357433 +0000 UTC m=+41.051631700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f18ecc2-6ee2-47fa-befa-83065346a118-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnx6b" (UID: "8f18ecc2-6ee2-47fa-befa-83065346a118") : secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:18.031510 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:18.031409    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 22:14:18.031510 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:18.031457    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-rbac-proxy-cm podName:8f18ecc2-6ee2-47fa-befa-83065346a118 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:22.031442612 +0000 UTC m=+41.051716891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnx6b" (UID: "8f18ecc2-6ee2-47fa-befa-83065346a118") : configmap "kube-rbac-proxy" not found
Oct 29 22:14:18.233864 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:18.233800    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:18.234044 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:18.233947    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:18.234044 ip-10-0-138-2 kubenswrapper[2572]: E1029 22:14:18.234015    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls podName:d4a55136-e7f7-46ac-b58e-24e95ab0aea8 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:26.23400036 +0000 UTC m=+45.254274611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls") pod "dns-default-xp49w" (UID: "d4a55136-e7f7-46ac-b58e-24e95ab0aea8") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:18.810328 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:18.810231    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v552v_1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:18.845734 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:18.845694    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hc4lm-rqsvk" event={"ID":"807ed4de-272f-4bb2-bf26-cc33f6967771","Type":"ContainerStarted","Data":"6c4cf8f1da814c401541accca948236cc4902ac0127d909c7e8595b448c3f902"}
Oct 29 22:14:18.886218 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:18.886161    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-hc4lm-rqsvk" podStartSLOduration=1.265208855 podStartE2EDuration="4.886141432s" totalImagesPullingTime="3.620932577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:18.88561588 +0000 UTC m=+37.905890148" watchObservedRunningTime="2026-10-29 22:14:18.886141432 +0000 UTC m=+37.906415709"
Oct 29 22:14:18.886369 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:18.886273    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5774f" podStartSLOduration=2.919188335 podStartE2EDuration="4.886266194s" totalImagesPullingTime="1.967077859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:17.883512338 +0000 UTC m=+36.903786612" watchObservedRunningTime="2026-10-29 22:14:18.886266194 +0000 UTC m=+37.906540470"
Oct 29 22:14:19.851678 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:19.851624    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg" event={"ID":"3104d9ac-59f1-406b-9d63-7af7892b40bf","Type":"ContainerStarted","Data":"b96699d7485b062c89c9feda449b44b42e716842336afa30b1ba91b8fce2925f"}
Oct 29 22:14:19.853447 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:19.853408    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5pq4b" event={"ID":"c03b55ec-8ed6-42d6-895c-2e7213a2ba39","Type":"ContainerStarted","Data":"eaf7b94b6080101303fd680a174f18dd82a1b1b531dad06a063fea3fe6df1349"}
Oct 29 22:14:19.895449 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:19.895386    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg" podStartSLOduration=1.851789375 podStartE2EDuration="4.895368072s" totalImagesPullingTime="3.043578697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:19.893893476 +0000 UTC m=+38.914167779" watchObservedRunningTime="2026-10-29 22:14:19.895368072 +0000 UTC m=+38.915642354"
Oct 29 22:14:19.933018 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:19.932958    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5pq4b" podStartSLOduration=33.424092088 podStartE2EDuration="37.932939977s" totalImagesPullingTime="4.508847889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:19.93205872 +0000 UTC m=+38.952332990" watchObservedRunningTime="2026-10-29 22:14:19.932939977 +0000 UTC m=+38.953214252"
Oct 29 22:14:20.858433 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:20.858373    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p5bdf" event={"ID":"02990faa-cba3-4132-9ca7-fe0f7202bbe5","Type":"ContainerStarted","Data":"cd987dcc2d3297c7b94181425c703f8305f64780d7975fea098a36226b660e39"}
Oct 29 22:14:20.858893 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:20.858702    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:14:20.904885 ip-10-0-138-2 kubenswrapper[2572]: I1029 22:14:20.904801    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-p5bdf" podStartSLOduration=32.877231554 podStartE2EDuration="38.904786124s" totalImagesPullingTime="6.02755457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:20.904512986 +0000 UTC m=+39.924787275" watchObservedRunningTime="2026-10-29 22:14:20.904786124 +0000 UTC m=+39.925060398"
Oct 29 22:14:21.016304 ip-10-0-138-2 systemd[1]: Stopping Kubernetes Kubelet...
Oct 29 22:14:21.029219 ip-10-0-138-2 systemd[1]: kubelet.service: Deactivated successfully.
Oct 29 22:14:21.029442 ip-10-0-138-2 systemd[1]: Stopped Kubernetes Kubelet.
Oct 29 22:14:21.029506 ip-10-0-138-2 systemd[1]: kubelet.service: Consumed 2.523s CPU time.
Oct 29 22:14:21.040520 ip-10-0-138-2 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 22:14:21.140015 ip-10-0-138-2 kubenswrapper[4559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:14:21.140015 ip-10-0-138-2 kubenswrapper[4559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 22:14:21.140015 ip-10-0-138-2 kubenswrapper[4559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:14:21.140015 ip-10-0-138-2 kubenswrapper[4559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:14:21.142165 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142078    4559 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:21.142165 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142162    4559 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:21.142165 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142165    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:21.142165 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142169    4559 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:21.142165 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142171    4559 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142175    4559 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142178    4559 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142188    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142191    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142194    4559 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142197    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142199    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142202    4559 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142205    4559 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142207    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142210    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142212    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142215    4559 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142217    4559 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142220    4559 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142222    4559 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142225    4559 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142227    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142230    4559 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:21.142341 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142232    4559 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142235    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142237    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142244    4559 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142246    4559 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142249    4559 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142251    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142254    4559 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142257    4559 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142260    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142262    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142264    4559 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142267    4559 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142269    4559 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142272    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142275    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142299    4559 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142304    4559 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142307    4559 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142310    4559 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:21.142881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142313    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142316    4559 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142318    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142321    4559 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142324    4559 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142326    4559 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142328    4559 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142332    4559 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142337    4559 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142341    4559 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142344    4559 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142347    4559 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142349    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142352    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142354    4559 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142358    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142361    4559 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142364    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142366    4559 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:21.143429 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142369    4559 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142371    4559 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142374    4559 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142376    4559 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142379    4559 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142381    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142384    4559 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142387    4559 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142389    4559 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142392    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142395    4559 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142397    4559 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142400    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142402    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142405    4559 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142407    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142410    4559 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142412    4559 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142415    4559 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:21.143930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142417    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142419    4559 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142423    4559 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142427    4559 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142430    4559 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142433    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142435    4559 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142438    4559 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142440    4559 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142445    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142447    4559 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142450    4559 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142452    4559 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142455    4559 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142457    4559 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142460    4559 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142462    4559 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142464    4559 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142467    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:21.144397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142469    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142473    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142475    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142478    4559 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.142482    4559 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143252    4559 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143260    4559 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143264    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143267    4559 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143270    4559 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143272    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143275    4559 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143278    4559 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143280    4559 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143283    4559 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143285    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143288    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143291    4559 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143294    4559 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:21.144911 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143296    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143299    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143301    4559 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143306    4559 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143309    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143313    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143315    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143318    4559 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143320    4559 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143323    4559 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143325    4559 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143328    4559 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143330    4559 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143333    4559 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143335    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143338    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143340    4559 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143343    4559 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143346    4559 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143349    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:21.145396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143352    4559 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143354    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143357    4559 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143359    4559 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143362    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143364    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143366    4559 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143369    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143371    4559 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143374    4559 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143376    4559 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143379    4559 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143381    4559 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143384    4559 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143386    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143390    4559 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143395    4559 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143399    4559 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143402    4559 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:21.145923 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143405    4559 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143408    4559 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143411    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143413    4559 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143415    4559 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143418    4559 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143420    4559 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143423    4559 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143426    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143428    4559 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143431    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143433    4559 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143436    4559 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143439    4559 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143442    4559 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143444    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143447    4559 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143449    4559 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143452    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143455    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:21.146397 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143458    4559 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143461    4559 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143464    4559 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143466    4559 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143468    4559 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143471    4559 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143473    4559 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143476    4559 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143489    4559 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143492    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143495    4559 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143498    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143500    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143503    4559 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143505    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143508    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143511    4559 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143513    4559 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143516    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143518    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:21.146913 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143521    4559 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143523    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143526    4559 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143528    4559 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143531    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143535    4559 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143538    4559 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143540    4559 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143543    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143545    4559 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143548    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143550    4559 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.143555    4559 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144120    4559 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144127    4559 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144132    4559 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144136    4559 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144139    4559 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144142    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:21.147407 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144145    4559 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144149    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144152    4559 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144154    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144157    4559 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144160    4559 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144162    4559 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144165    4559 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144167    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144169    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144172    4559 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144174    4559 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144177    4559 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144180    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144182    4559 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144186    4559 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144190    4559 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144193    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144197    4559 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:21.147910 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144200    4559 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144203    4559 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144205    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144208    4559 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144210    4559 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144213    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144215    4559 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144218    4559 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144220    4559 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144223    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144225    4559 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144228    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144230    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144233    4559 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144237    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144240    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144242    4559 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144245    4559 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144247    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:21.148639 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144250    4559 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144252    4559 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144255    4559 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144257    4559 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144259    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144262    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144265    4559 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144267    4559 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144269    4559 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144272    4559 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144275    4559 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144277    4559 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144279    4559 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144282    4559 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144285    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144287    4559 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144290    4559 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144292    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144295    4559 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144297    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:21.149541 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144299    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144302    4559 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144305    4559 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144307    4559 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144310    4559 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144313    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144315    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144319    4559 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144321    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144324    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144326    4559 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144329    4559 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144331    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144334    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144336    4559 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144338    4559 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144341    4559 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144344    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144347    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:21.150611 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144350    4559 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144352    4559 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144355    4559 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144357    4559 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144359    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144362    4559 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144364    4559 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144368    4559 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144370    4559 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144373    4559 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144375    4559 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144378    4559 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144380    4559 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144383    4559 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144385    4559 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144388    4559 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144391    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144393    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144395    4559 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144398    4559 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:21.151396 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144403    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144405    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.144409    4559 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144495    4559 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144502    4559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144512    4559 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144519    4559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144526    4559 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144531    4559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144538    4559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144544    4559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144547    4559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144550    4559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144555    4559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144559    4559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144562    4559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144565    4559 flags.go:64] FLAG: --cgroup-root=""
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144568    4559 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144571    4559 flags.go:64] FLAG: --client-ca-file=""
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144574    4559 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144578    4559 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144582    4559 flags.go:64] FLAG: --cluster-domain=""
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144585    4559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144588    4559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 22:14:21.152037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144591    4559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144594    4559 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144599    4559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144601    4559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144604    4559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144608    4559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144611    4559 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144614    4559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144617    4559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144621    4559 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144623    4559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144628    4559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144631    4559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144634    4559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144637    4559 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144639    4559 flags.go:64] FLAG: --enable-server="true"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144642    4559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144647    4559 flags.go:64] FLAG: --event-burst="100"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144650    4559 flags.go:64] FLAG: --event-qps="50"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144652    4559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144655    4559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144659    4559 flags.go:64] FLAG: --eviction-hard=""
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144663    4559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144666    4559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144669    4559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 22:14:21.152731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144672    4559 flags.go:64] FLAG: --eviction-soft=""
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144675    4559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144677    4559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144680    4559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144684    4559 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144687    4559 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144690    4559 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144692    4559 flags.go:64] FLAG: --feature-gates=""
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144696    4559 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144699    4559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144702    4559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144705    4559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144708    4559 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144711    4559 flags.go:64] FLAG: --help="false"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144714    4559 flags.go:64] FLAG: --hostname-override="ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144717    4559 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144720    4559 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144723    4559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144727    4559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144730    4559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144733    4559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144736    4559 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144739    4559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 22:14:21.153416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144741    4559 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144744    4559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144747    4559 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144750    4559 flags.go:64] FLAG: --kube-reserved=""
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144753    4559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144760    4559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144764    4559 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144766    4559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144769    4559 flags.go:64] FLAG: --lock-file=""
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144772    4559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144775    4559 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144778    4559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144784    4559 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144787    4559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144794    4559 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144797    4559 flags.go:64] FLAG: --logging-format="text"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144800    4559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144803    4559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144820    4559 flags.go:64] FLAG: --manifest-url=""
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144823    4559 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144828    4559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144831    4559 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144835    4559 flags.go:64] FLAG: --max-pods="110"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144838    4559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144841    4559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 22:14:21.154013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144844    4559 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144847    4559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144849    4559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144852    4559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144855    4559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144862    4559 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144865    4559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144869    4559 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144872    4559 flags.go:64] FLAG: --pod-cidr=""
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144874    4559 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144877    4559 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144880    4559 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144884    4559 flags.go:64] FLAG: --port="10250"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144887    4559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144889    4559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b5bb37fcb252ca55"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144892    4559 flags.go:64] FLAG: --qos-reserved=""
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144895    4559 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144898    4559 flags.go:64] FLAG: --register-node="true"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144901    4559 flags.go:64] FLAG: --register-with-taints=""
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144905    4559 flags.go:64] FLAG: --registry-burst="10"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144907    4559 flags.go:64] FLAG: --registry-qps="5"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144910    4559 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144913    4559 flags.go:64] FLAG: --reserved-memory=""
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144917    4559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144921    4559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144923    4559 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 22:14:21.154620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144927    4559 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144930    4559 flags.go:64] FLAG: --runonce="false"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144933    4559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144936    4559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144938    4559 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144941    4559 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144944    4559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144947    4559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144950    4559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144953    4559 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144956    4559 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144959    4559 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144962    4559 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144965    4559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144968    4559 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144971    4559 flags.go:64] FLAG: --system-cgroups=""
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144973    4559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144979    4559 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144983    4559 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144985    4559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144989    4559 flags.go:64] FLAG: --tls-min-version=""
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144992    4559 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144995    4559 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.144998    4559 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.145001    4559 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 22:14:21.155306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.145004    4559 flags.go:64] FLAG: --v="2"
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.145009    4559 flags.go:64] FLAG: --version="false"
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.145014    4559 flags.go:64] FLAG: --vmodule=""
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.145018    4559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.145022    4559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145153    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145157    4559 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145160    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145164    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145166    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145169    4559 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145172    4559 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145174    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145177    4559 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145179    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145182    4559 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145184    4559 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145187    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145190    4559 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145192    4559 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145194    4559 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:21.155985 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145197    4559 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145199    4559 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145202    4559 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145205    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145209    4559 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145213    4559 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145216    4559 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145220    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145222    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145226    4559 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145231    4559 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145234    4559 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145238    4559 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145241    4559 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145244    4559 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145247    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145249    4559 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145252    4559 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145255    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:21.156537 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145257    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145259    4559 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145262    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145264    4559 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145267    4559 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145269    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145272    4559 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145274    4559 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145277    4559 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145279    4559 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145282    4559 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145284    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145287    4559 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145289    4559 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145292    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145294    4559 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145297    4559 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145301    4559 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145303    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145306    4559 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:21.157019 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145308    4559 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145310    4559 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145313    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145315    4559 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145317    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145321    4559 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145323    4559 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145326    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145328    4559 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145331    4559 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145334    4559 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145337    4559 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145340    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145342    4559 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145344    4559 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145347    4559 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145349    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145352    4559 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145354    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145357    4559 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:21.157636 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145359    4559 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145362    4559 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145364    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145367    4559 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145369    4559 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145372    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145374    4559 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145376    4559 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145379    4559 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145383    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145386    4559 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145388    4559 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145391    4559 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145393    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145396    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145398    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145401    4559 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145405    4559 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145407    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:21.158274 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145410    4559 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:21.158769 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145412    4559 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:21.158769 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145415    4559 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:21.158769 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145418    4559 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:21.158769 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145420    4559 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:21.158769 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145423    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:21.158769 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145425    4559 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:21.158769 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145428    4559 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:21.158769 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145430    4559 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:21.158769 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145433    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:21.158769 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145435    4559 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:21.158769 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145437    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:21.158769 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.145462    4559 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:14:21.158769 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145504    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:21.158769 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145509    4559 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:21.158769 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145512    4559 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:21.158769 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145515    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145518    4559 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145520    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145523    4559 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145526    4559 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145528    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145533    4559 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145536    4559 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145538    4559 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145540    4559 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145543    4559 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145545    4559 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145548    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145551    4559 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145554    4559 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145559    4559 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145562    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145565    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145569    4559 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:21.159197 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145573    4559 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145575    4559 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145578    4559 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145581    4559 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145584    4559 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145586    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145588    4559 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145591    4559 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145593    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145596    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145598    4559 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145601    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145603    4559 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145606    4559 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145608    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145610    4559 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145613    4559 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145616    4559 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145618    4559 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145623    4559 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:21.159685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145625    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145627    4559 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145630    4559 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145632    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145635    4559 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145637    4559 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145640    4559 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145642    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145647    4559 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145650    4559 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145652    4559 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145655    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145657    4559 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145659    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145662    4559 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145665    4559 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145667    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145670    4559 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145672    4559 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145675    4559 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:21.160251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145677    4559 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145680    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145682    4559 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145685    4559 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145687    4559 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145689    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145692    4559 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145695    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145697    4559 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145699    4559 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145702    4559 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145706    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145709    4559 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145711    4559 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145714    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145716    4559 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145719    4559 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145721    4559 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145723    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:21.160779 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145726    4559 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145729    4559 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145732    4559 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145734    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145737    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145739    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145742    4559 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145744    4559 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145747    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145750    4559 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145752    4559 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145755    4559 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145757    4559 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145759    4559 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145762    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145765    4559 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145767    4559 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145770    4559 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145772    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145774    4559 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:21.161342 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145777    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145779    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145781    4559 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145784    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.145788    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.152146    4559 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.152163    4559 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.152187    4559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.152192    4559 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152251    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152257    4559 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152260    4559 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152262    4559 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152265    4559 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152268    4559 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152271    4559 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152274    4559 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152276    4559 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152279    4559 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152282    4559 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:21.161957 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152284    4559 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152287    4559 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152290    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152297    4559 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152300    4559 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152303    4559 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152306    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152308    4559 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152311    4559 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152313    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152316    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152319    4559 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152321    4559 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152324    4559 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152326    4559 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152329    4559 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152332    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152335    4559 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152338    4559 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:21.162501 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152341    4559 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152343    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152346    4559 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152349    4559 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152351    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152354    4559 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152358    4559 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152362    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152365    4559 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152368    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152372    4559 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152375    4559 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152378    4559 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152381    4559 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152383    4559 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152386    4559 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152393    4559 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152397    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152400    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:21.163044 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152402    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152405    4559 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152407    4559 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152410    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152412    4559 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152415    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152417    4559 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152420    4559 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152422    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152425    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152428    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152431    4559 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152433    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152436    4559 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152439    4559 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152441    4559 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152444    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152448    4559 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152452    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:21.163509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152454    4559 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152457    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152460    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152462    4559 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152465    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152467    4559 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152470    4559 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152472    4559 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152475    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152478    4559 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152480    4559 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152487    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152490    4559 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152493    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152495    4559 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152498    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152500    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152503    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152505    4559 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152508    4559 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:21.163993 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152510    4559 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:21.164489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152513    4559 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:21.164489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152516    4559 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:21.164489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152518    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:21.164489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152521    4559 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:21.164489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152523    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:21.164489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152526    4559 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:21.164489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152528    4559 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:21.164489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152531    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:21.164489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152533    4559 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:21.164489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152536    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:21.164489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152539    4559 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:21.164489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152541    4559 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:21.164489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152544    4559 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:21.164489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152546    4559 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:21.164489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152549    4559 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:21.164489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152552    4559 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:21.164489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152554    4559 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:21.164930 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.152583    4559 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:14:21.164930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152743    4559 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:21.164930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152747    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:21.164930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152750    4559 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:21.164930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152753    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:21.164930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152755    4559 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:21.164930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152763    4559 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:21.164930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152766    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:21.164930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152769    4559 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:21.164930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152771    4559 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:21.164930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152774    4559 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:21.164930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152776    4559 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:21.164930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152778    4559 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:21.164930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152781    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:21.164930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152783    4559 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:21.164930 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152786    4559 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152788    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152791    4559 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152794    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152797    4559 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152799    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152802    4559 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152804    4559 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152822    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152824    4559 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152827    4559 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152830    4559 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152832    4559 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152835    4559 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152837    4559 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152840    4559 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152842    4559 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152845    4559 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152847    4559 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:21.165322 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152850    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152852    4559 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152855    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152858    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152861    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152869    4559 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152872    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152874    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152876    4559 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152879    4559 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152881    4559 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152884    4559 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152886    4559 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152889    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152891    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152894    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152896    4559 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152899    4559 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152901    4559 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:21.165777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152904    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152906    4559 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152908    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152911    4559 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152914    4559 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152916    4559 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152919    4559 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152921    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152925    4559 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152929    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152932    4559 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152935    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152938    4559 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152941    4559 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152943    4559 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152947    4559 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152952    4559 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152954    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152957    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:21.166356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152966    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152969    4559 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152971    4559 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152973    4559 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152976    4559 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152979    4559 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152981    4559 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152984    4559 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152987    4559 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152989    4559 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152991    4559 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152994    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152996    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.152999    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153001    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153003    4559 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153006    4559 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153009    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153011    4559 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:21.166824 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153014    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:21.167281 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153016    4559 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:21.167281 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153019    4559 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:21.167281 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153021    4559 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:21.167281 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153024    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:21.167281 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153026    4559 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:21.167281 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153028    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:21.167281 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153031    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:21.167281 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153034    4559 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:21.167281 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153036    4559 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:21.167281 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153038    4559 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:21.167281 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153041    4559 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:21.167281 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153044    4559 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:21.167281 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153046    4559 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:21.167281 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153055    4559 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:21.167281 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153057    4559 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:21.167281 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.153081    4559 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153119    4559 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153123    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153127    4559 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153131    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153134    4559 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153137    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153140    4559 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153142    4559 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153145    4559 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153149    4559 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153152    4559 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153155    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153157    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153159    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153162    4559 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153164    4559 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153167    4559 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153170    4559 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:21.168963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153172    4559 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153175    4559 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153178    4559 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153180    4559 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153183    4559 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153185    4559 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153188    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153190    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153193    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153196    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153198    4559 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153201    4559 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153209    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153212    4559 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153214    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153217    4559 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153219    4559 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153222    4559 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153224    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:21.169430 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153227    4559 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153229    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153232    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153235    4559 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153237    4559 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153240    4559 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153242    4559 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153245    4559 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153247    4559 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153250    4559 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153252    4559 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153255    4559 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153257    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153260    4559 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153262    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153265    4559 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153270    4559 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153273    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153275    4559 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153278    4559 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:21.169908 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153280    4559 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153282    4559 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153285    4559 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153287    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153289    4559 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153292    4559 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153300    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153302    4559 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153305    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153307    4559 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153310    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153312    4559 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153315    4559 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153317    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153319    4559 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153322    4559 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153324    4559 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153326    4559 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153329    4559 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:21.170437 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153331    4559 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153333    4559 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153336    4559 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153339    4559 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153341    4559 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153343    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153346    4559 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153348    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153351    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153357    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153360    4559 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153362    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153365    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153367    4559 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153370    4559 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153372    4559 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153375    4559 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153377    4559 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153379    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:21.170915 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153382    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:21.171373 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153390    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:21.171373 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153393    4559 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:21.171373 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153395    4559 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:21.171373 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153398    4559 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:21.171373 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153400    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:21.171373 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153402    4559 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:21.171373 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153405    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:21.171373 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153408    4559 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:21.171373 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153410    4559 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:21.171373 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:21.153413    4559 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:21.171373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.153493    4559 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 22:14:21.171373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.154514    4559 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 29 22:14:21.171373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.154578    4559 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 29 22:14:21.171373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.154798    4559 server.go:1014] "Starting client certificate rotation"
Oct 29 22:14:21.171373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.154938    4559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:14:21.171373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.154986    4559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:08:41 +0000 UTC" deadline="2028-07-04 05:31:33.630320217 +0000 UTC"
Oct 29 22:14:21.171373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.155028    4559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14719h17m12.475293973s"
Oct 29 22:14:21.171858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.155302    4559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:14:21.171858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.156431    4559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:14:21.171858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.157895    4559 log.go:25] "Validated CRI v1 runtime API"
Oct 29 22:14:21.171858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.159451    4559 log.go:25] "Validated CRI v1 image API"
Oct 29 22:14:21.171858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.160165    4559 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 22:14:21.171858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.161772    4559 fs.go:135] Filesystem UUIDs: map[52c1ed33-7d7f-4680-9bf8-d4a899aa2f2c:/dev/nvme0n1p4 66777707-1b6e-4df8-8917-8ae6b2af411b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 29 22:14:21.172027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.161796    4559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/044bc26e92f3f0b48c48ddfb35af3edabddcd9e052b53d32258b34a2bcf45694/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/044bc26e92f3f0b48c48ddfb35af3edabddcd9e052b53d32258b34a2bcf45694/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09ec796e175c68a7ecf208e3fd25136253c240fb603980b9d7cf51f6c295cad2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09ec796e175c68a7ecf208e3fd25136253c240fb603980b9d7cf51f6c295cad2/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0de9b0e763875276bbefb78c429db0f6b6a1d9165ceba2b1e61e6841cf2b46c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0de9b0e763875276bbefb78c429db0f6b6a1d9165ceba2b1e61e6841cf2b46c5/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10b41b6adef4be966c582cdc48e40fadb28dd646302d456467848faed34ec445/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10b41b6adef4be966c582cdc48e40fadb28dd646302d456467848faed34ec445/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/408002ab0d76e5b8292d2f62e2ebbbf8b28d0c79d00208c735d954af57dc27b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/408002ab0d76e5b8292d2f62e2ebbbf8b28d0c79d00208c735d954af57dc27b5/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4750ae6c337a571a725edd2562a3517a062504a3ade721861ff54c1364a5675a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4750ae6c337a571a725edd2562a3517a062504a3ade721861ff54c1364a5675a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e00e456a78e0e6d254bc031812ca35187fdb18b65303fe01df2736f70ebcb99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e00e456a78e0e6d254bc031812ca35187fdb18b65303fe01df2736f70ebcb99/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b9a07980d4585ffee63d09c704ea6c68527a26286578c28d44f244aaf634d12/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b9a07980d4585ffee63d09c704ea6c68527a26286578c28d44f244aaf634d12/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d7c43ba30f9c0725e2136f33a7795b1e1bb07bdd28c8c1a1ebbd0cf9bd9e646/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d7c43ba30f9c0725e2136f33a7795b1e1bb07bdd28c8c1a1ebbd0cf9bd9e646/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7ddd0d866bb33c277ffe06ba1ea76a766bc499124e08a77ebfdf1a4d28ba0d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7ddd0d866bb33c277ffe06ba1ea76a766bc499124e08a77ebfdf1a4d28ba0d5/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4d57d45e00a8156835d17e56224cc17abe814c33632ca86409e4b9a6a9ddcfc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4d57d45e00a8156835d17e56224cc17abe814c33632ca86409e4b9a6a9ddcfc/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5c5bbf24408fd6a752a915c8039399c0009aee67effacb2986761558aef87bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5c5bbf24408fd6a752a915c8039399c0009aee67effacb2986761558aef87bf/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7dc3bc4c972c100a2074f4299f4962f9febc7bc6a8bdb9458e79686fd5c0742/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7dc3bc4c972c100a2074f4299f4962f9febc7bc6a8bdb9458e79686fd5c0742/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cca931e0b108a89e3426252e0979e7dd87f22129f2bb5578ef6903f5ea74521e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cca931e0b108a89e3426252e0979e7dd87f22129f2bb5578ef6903f5ea74521e/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de3a0df9870dc7439e231f189c64c887a86943e666eeae14f03380d0301dbf89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de3a0df9870dc7439e231f189c64c887a86943e666eeae14f03380d0301dbf89/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e075d51ca14e4f7237ba867d6845ddcd67c963a12ee720c3c8786c47ebcebac9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e075d51ca14e4f7237ba867d6845ddcd67c963a12ee720c3c8786c47ebcebac9/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02990faa-cba3-4132-9ca7-fe0f7202bbe5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/02990faa-cba3-4132-9ca7-fe0f7202bbe5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/198638bc-ff2e-4c81-a599-616e7b015749/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/198638bc-ff2e-4c81-a599-616e7b015749/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/198638bc-ff2e-4c81-a599-616e7b015749/volumes/kubernetes.io~projected/kube-api-access-ml2q5:{mountpoint:/var/lib/kubelet/pods/198638bc-ff2e-4c81-a599-616e7b015749/volumes/kubernetes.io~projected/kube-api-access-ml2q5 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/198638bc-ff2e-4c81-a599-616e7b015749/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/198638bc-ff2e-4c81-a599-616e7b015749/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23921851-35c8-450c-a184-fe5ed0b1c6a7/volumes/kubernetes.io~projected/kube-api-access-x2bf4:{mountpoint:/var/lib/kubelet/pods/23921851-35c8-450c-a184-fe5ed0b1c6a7/volumes/kubernetes.io~projected/kube-api-access-x2bf4 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3104d9ac-59f1-406b-9d63-7af7892b40bf/volumes/kubernetes.io~projected/kube-api-access-5qv6v:{mountpoint:/var/lib/kubelet/pods/3104d9ac-59f1-406b-9d63-7af7892b40bf/volumes/kubernetes.io~projected/kube-api-access-5qv6v major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3104d9ac-59f1-406b-9d63-7af7892b40bf/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/3104d9ac-59f1-406b-9d63-7af7892b40bf/volumes/kubernetes.io~secret/signing-key major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37a98266-fad3-440d-95d4-85de9ad6f476/volumes/kubernetes.io~projected/kube-api-access-n27db:{mountpoint:/var/lib/kubelet/pods/37a98266-fad3-440d-95d4-85de9ad6f476/volumes/kubernetes.io~projected/kube-api-access-n27db major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/486bbe97-6300-49e9-a0f3-c9d316333048/volumes/kubernetes.io~projected/kube-api-access-lxlcz:{mountpoint:/var/lib/kubelet/pods/486bbe97-6300-49e9-a0f3-c9d316333048/volumes/kubernetes.io~projected/kube-api-access-lxlcz major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/807ed4de-272f-4bb2-bf26-cc33f6967771/volumes/kubernetes.io~projected/kube-api-access-kq8zc:{mountpoint:/var/lib/kubelet/pods/807ed4de-272f-4bb2-bf26-cc33f6967771/volumes/kubernetes.io~projected/kube-api-access-kq8zc major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f18ecc2-6ee2-47fa-befa-83065346a118/volumes/kubernetes.io~projected/kube-api-access-qclfj:{mountpoint:/var/lib/kubelet/pods/8f18ecc2-6ee2-47fa-befa-83065346a118/volumes/kubernetes.io~projected/kube-api-access-qclfj major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b3ec29f-f4e7-4811-83ab-50680e0e60e7/volumes/kubernetes.io~projected/kube-api-access-xsrvw:{mountpoint:/var/lib/kubelet/pods/9b3ec29f-f4e7-4811-83ab-50680e0e60e7/volumes/kubernetes.io~projected/kube-api-access-xsrvw major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de/volumes/kubernetes.io~projected/kube-api-access-pbch6:{mountpoint:/var/lib/kubelet/pods/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de/volumes/kubernetes.io~projected/kube-api-access-pbch6 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c03b55ec-8ed6-42d6-895c-2e7213a2ba39/volumes/kubernetes.io~projected/kube-api-access-bvr8w:{mountpoint:/var/lib/kubelet/pods/c03b55ec-8ed6-42d6-895c-2e7213a2ba39/volumes/kubernetes.io~projected/kube-api-access-bvr8w major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55/volumes/kubernetes.io~projected/kube-api-access-vsljd:{mountpoint:/var/lib/kubelet/pods/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55/volumes/kubernetes.io~projected/kube-api-access-vsljd major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc8f498e-4039-43ef-91f7-036179511f01/volumes/kubernetes.io~projected/kube-api-access-ffhsv:{mountpoint:/var/lib/kubelet/pods/cc8f498e-4039-43ef-91f7-036179511f01/volumes/kubernetes.io~projected/kube-api-access-ffhsv major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4a55136-e7f7-46ac-b58e-24e95ab0aea8/volumes/kubernetes.io~projected/kube-api-access-5blnb:{mountpoint:/var/lib/kubelet/pods/d4a55136-e7f7-46ac-b58e-24e95ab0aea8/volumes/kubernetes.io~projected/kube-api-access-5blnb major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d637bbed-2662-4313-9639-e7a3bc148955/volumes/kubernetes.io~projected/kube-api-access-zm7gg:{mountpoint:/var/lib/kubelet/pods/d637bbed-2662-4313-9639-e7a3bc148955/volumes/kubernetes.io~projected/kube-api-access-zm7gg major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d637bbed-2662-4313-9639-e7a3bc148955/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/d637bbed-2662-4313-9639-e7a3bc148955/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3c107fc-072e-45ce-b4cd-acbe1284be88/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f3c107fc-072e-45ce-b4cd-acbe1284be88/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3c107fc-072e-45ce-b4cd-acbe1284be88/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f3c107fc-072e-45ce-b4cd-acbe1284be88/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3c107fc-072e-45ce-b4cd-acbe1284be88/volumes/kubernetes.io~projected/kube-api-access-m8cpn:{mountpoint:/var/lib/kubelet/pods/f3c107fc-072e-45ce-b4cd-acbe1284be88/volumes/kubernetes.io~projected/kube-api-access-m8cpn major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5cf9217-fa53-45ec-b7fa-a80520684473/volumes/kubernetes.io~projected/kube-api-access-gbxms:{mountpoint:/var/lib/kubelet/pods/f5cf9217-fa53-45ec-b7fa-a80520684473/volumes/kubernetes.io~projected/kube-api-access-gbxms major:0 minor:226 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c7f393c879364caa5320027067c61b43fe65ce51d6cf50ac00ce0df7f7155520/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/05ae6afd4f8e8dddc5d1fc68ddd79e3bbc892fc8d552050a60841a938c9e4713/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/db034dc00981351240fda49e95566c08c26df801b49c46daced5f6fc485ea558/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/57372c97debf9a666cc68e679c3648c174a4cd90fd53f9048fbda463a6758f71/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/721f4eb4c352080f97e80d1fcb8395ba4a40ae4b8b2e7494a99e9472e9397c39/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d1c7c021ec9a8cca302afe8bbb68dfd8e9a437d8280335e0e0e5d9f73c46bbd1/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/68bba9f1cfa047ae40d3fcaffc70e96060c6000f274b9fe1a4e8f301e6f61220/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/51be51435ae1a9306bc7275dc299c46f435b2cda6e3901e66196f9c8124c2a78/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b8c0b589502c6a73b1dcd0870c2d4a77bca8398882b2e1e56e651bfd5a77d687/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/261a436c28bfe427d87e65378bccd71f663c69f5363a816be22e19cd33c90ed7/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/1798653f8df8fd553ca41e565a2eaf88c30f1b5aa9255202e79358c04970bb26/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/8797750680cf713d15f67802eae4b29252078f01ec7e9768f526dd64f527be9d/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/5449fee4aab2197d2d28e8ce301952c5624619d12aa2c651fa4d7d5319967b6f/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/275677df5d9f46fccc5dc4114493cf118492b92e92cda02b363bfe7c361212d4/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d72f7146098dee101d20c1aeab63c167c81222955801ef77f17176dd032aa127/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6309070ac33ee6e6ef3e6c897a57b9b3303d10302f350e0b28b7b20b454ab00a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/04a87e0a390ad54fcf00aa669b45bfafd6025b9e06ea2c5cdf137cc1d0b18ce8/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/02c7a925303d27705c73d64bffbf17e933a48622894062393d74b5fe55ba7d10/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d82e4c2bc49e6eeaf80dccdb6afb610f58b76b4899d479cc8d7e4bd3ba85d8e2/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a0b8be0a24747ee92e04ddb7f9acd982dad2b8c8bf1e7180dc800cfda50a74a5/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/0686fc55db2c078534bc184df71fbeeeb301c3244298ef8218a518e64d4eaaa0/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/29ea908015efad61842214d8f7615a9195a5781a1fad8ccdc3431ac7edcc0ed0/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/91dcd0eb432e6f28453432508ea51ea12521a504b5afc70dca2beae6445f0fa9/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/513ce13873e5394c055a2897abadd9d7a2cb70220cd6c5228665888515e23713/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/e3ef09680c7decef8ca6db78e07e91b00bb091c7bf6f5e5a7989c13b8a3ede66/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/9f0f060f0ccd8ff467606a4072cd6b89f281d1cf5095fb15c8589be33c89d496/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/86b057ddf79b480cc45b4f965fc23131259a967fb865e621b53230050235decc/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/db48449ec121e561e20af6ebec0397dae6a28a48a55ed2091b7f50da445cd709/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/4f20745caacc27a272b639c7cdc5f21de2df057595b8d172d01886a2d2f2350d/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/dc6538dc206bfe3fb12d74228a042fc660676a7a3d58aa05b6e49d4e17c05f04/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/5609b6575336d5e62e751ab0b903405b17408a4319d9870a265d2aca24829ef8/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/aeaeedf262ad9c137e870619961f13cd0bc0d1d2f275d0a811ce1ff5f8ce3b73/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/a02fb8dd4a2cf8ba8851fb7d9d07531efa0e5df73876231b1f214b50d34d3b54/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/51d60fcdaa0e2f0b5a30200916cb9b7a6a4241174401940ad7c39e89f8dc34d0/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/9f528448285d54589483fb4c9073df77cc9bbab5532acacc3d6c3204ba6903fe/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/86e489e9076341a50930eb315c54234c2e8e986a9893afe7723cbc04df889829/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/18bd5d6f7f082c8b0b91da62927eeb0e331deb26946e4696544fb23086054ee4/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/2292ef4161e3cf5fc3936860323d9458b17a0201ad9b5b72398e44f01eb3446a/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/f1c28bd4f572ec4c00121ec34f14eba15905d43920ac7499af0afa2d51e12c8e/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/3ab0bc1f2544162ffcc8daf40ff50d71c45db95b3752f51fb8f4d25e4a28f8e0/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/53b5a5ff37feebd69cb60bf655e053191229e292f554e0de9ed139ad643669d4/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/cd45413045592dfe4976b851c011ea91207227c436490dd84d0cb555f7ff1521/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/5ad00781756f68da08c928547551f6d42648f8136bebb9c10261f000f8539f6d/merged major:0 minor:97 fsType:overlay blockSize:0}]
Oct 29 22:14:21.172170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.170093    4559 manager.go:217] Machine: {Timestamp:2026-10-29 22:14:21.169606988 +0000 UTC m=+0.081050532 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098786 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20990956dce6d87ddc7c4da222c7e7 SystemUUID:ec209909-56dc-e6d8-7ddc-7c4da222c7e7 BootID:45cbb238-5b03-4856-83d8-41c35a7f6e26 Filesystems:[{Device:/run/containers/storage/overlay-containers/09ec796e175c68a7ecf208e3fd25136253c240fb603980b9d7cf51f6c295cad2/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0de9b0e763875276bbefb78c429db0f6b6a1d9165ceba2b1e61e6841cf2b46c5/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d637bbed-2662-4313-9639-e7a3bc148955/volumes/kubernetes.io~projected/kube-api-access-zm7gg DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f5cf9217-fa53-45ec-b7fa-a80520684473/volumes/kubernetes.io~projected/kube-api-access-gbxms DeviceMajor:0 DeviceMinor:226 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/37a98266-fad3-440d-95d4-85de9ad6f476/volumes/kubernetes.io~projected/kube-api-access-n27db DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b9a07980d4585ffee63d09c704ea6c68527a26286578c28d44f244aaf634d12/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de/volumes/kubernetes.io~projected/kube-api-access-pbch6 DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55/volumes/kubernetes.io~projected/kube-api-access-vsljd DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7dc3bc4c972c100a2074f4299f4962f9febc7bc6a8bdb9458e79686fd5c0742/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f3c107fc-072e-45ce-b4cd-acbe1284be88/volumes/kubernetes.io~projected/kube-api-access-m8cpn DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/23921851-35c8-450c-a184-fe5ed0b1c6a7/volumes/kubernetes.io~projected/kube-api-access-x2bf4 DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cca931e0b108a89e3426252e0979e7dd87f22129f2bb5578ef6903f5ea74521e/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9b3ec29f-f4e7-4811-83ab-50680e0e60e7/volumes/kubernetes.io~projected/kube-api-access-xsrvw DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/198638bc-ff2e-4c81-a599-616e7b015749/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5c5bbf24408fd6a752a915c8039399c0009aee67effacb2986761558aef87bf/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f18ecc2-6ee2-47fa-befa-83065346a118/volumes/kubernetes.io~projected/kube-api-access-qclfj DeviceMajor:0 DeviceMinor:218 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/198638bc-ff2e-4c81-a599-616e7b015749/volumes/kubernetes.io~projected/kube-api-access-ml2q5 DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d7c43ba30f9c0725e2136f33a7795b1e1bb07bdd28c8c1a1ebbd0cf9bd9e646/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3104d9ac-59f1-406b-9d63-7af7892b40bf/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:233 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d637bbed-2662-4313-9639-e7a3bc148955/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c03b55ec-8ed6-42d6-895c-2e7213a2ba39/volumes/kubernetes.io~projected/kube-api-access-bvr8w DeviceMajor:0 DeviceMinor:213 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc8f498e-4039-43ef-91f7-036179511f01/volumes/kubernetes.io~projected/kube-api-access-ffhsv DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e075d51ca14e4f7237ba867d6845ddcd67c963a12ee720c3c8786c47ebcebac9/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3104d9ac-59f1-406b-9d63-7af7892b40bf/volumes/kubernetes.io~projected/kube-api-access-5qv6v DeviceMajor:0 DeviceMinor:234 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f3c107fc-072e-45ce-b4cd-acbe1284be88/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/486bbe97-6300-49e9-a0f3-c9d316333048/volumes/kubernetes.io~projected/kube-api-access-lxlcz DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4d57d45e00a8156835d17e56224cc17abe814c33632ca86409e4b9a6a9ddcfc/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/198638bc-ff2e-4c81-a599-616e7b015749/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4750ae6c337a571a725edd2562a3517a062504a3ade721861ff54c1364a5675a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/408002ab0d76e5b8292d2f62e2ebbbf8b28d0c79d00208c735d954af57dc27b5/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/02990faa-cba3-4132-9ca7-fe0f7202bbe5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/f3c107fc-072e-45ce-b4cd-acbe1284be88/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e00e456a78e0e6d254bc031812ca35187fdb18b65303fe01df2736f70ebcb99/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/044bc26e92f3f0b48c48ddfb35af3edabddcd9e052b53d32258b34a2bcf45694/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de3a0df9870dc7439e231f189c64c887a86943e666eeae14f03380d0301dbf89/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/d4a55136-e7f7-46ac-b58e-24e95ab0aea8/volumes/kubernetes.io~projected/kube-api-access-5blnb DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10b41b6adef4be966c582cdc48e40fadb28dd646302d456467848faed34ec445/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/807ed4de-272f-4bb2-bf26-cc33f6967771/volumes/kubernetes.io~projected/kube-api-access-kq8zc DeviceMajor:0 DeviceMinor:219 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7ddd0d866bb33c277ffe06ba1ea76a766bc499124e08a77ebfdf1a4d28ba0d5/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:8d7c43ba30f9c07 MacAddress:2a:ca:b6:f8:d6:02 Speed:10000 Mtu:8901} {Name:b7ddd0d866bb33c MacAddress:82:46:28:d0:ea:9b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fe:40:f1:3b Speed:0 Mtu:9001} {Name:br-int MacAddress:ae:ed:98:a4:7a:16 Speed:0 Mtu:8901} {Name:c4d57d45e00a815 MacAddress:62:b9:39:2d:21:82 Speed:10000 Mtu:8901} {Name:c5c5bbf24408fd6 MacAddress:76:b2:c0:b2:ae:66 Speed:10000 Mtu:8901} {Name:cca931e0b108a89 MacAddress:f2:a4:1f:a3:3a:1b Speed:10000 Mtu:8901} {Name:de3a0df9870dc74 MacAddress:da:ec:ab:6c:ad:8b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:fe:40:f1:3b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:b6:1d:57:1a:ce:05 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b2:50:2d:d7:3d:6b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 22:14:21.172170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.170392    4559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 22:14:21.172170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.170438    4559 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 22:14:21.172170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.170750    4559 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 22:14:21.172170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.170780    4559 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-2.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 22:14:21.172170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.170928    4559 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 22:14:21.172170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.170936    4559 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 22:14:21.172170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.170946    4559 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:14:21.172170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.170960    4559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:14:21.172170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.171058    4559 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 22:14:21.172170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.171228    4559 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 22:14:21.172170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.171344    4559 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 22:14:21.172170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.171386    4559 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 22:14:21.172170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.171402    4559 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 22:14:21.172170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.171412    4559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 22:14:21.172170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.171421    4559 kubelet.go:405] "Adding apiserver pod source"
Oct 29 22:14:21.172170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.171429    4559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 22:14:21.172732 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.172269    4559 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 22:14:21.172732 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.172282    4559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:14:21.172853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.172831    4559 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 22:14:21.174998 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.174985    4559 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 22:14:21.174998 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.174997    4559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:14:21.175109 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.175100    4559 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 29 22:14:21.175310 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.175299    4559 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 22:14:21.175346 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.175329    4559 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 22:14:21.175453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.175441    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 22:14:21.175512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.175461    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 22:14:21.175512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.175471    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 22:14:21.175512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.175477    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 22:14:21.175512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.175495    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 22:14:21.175512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.175503    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 22:14:21.175512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.175509    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 22:14:21.175512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.175514    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 22:14:21.175781 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.175521    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 22:14:21.175781 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.175527    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 22:14:21.175781 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.175536    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 22:14:21.175781 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.175545    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 22:14:21.175781 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.175569    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 22:14:21.175781 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.175574    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 22:14:21.178984 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.178971    4559 server.go:1263] "Started kubelet"
Oct 29 22:14:21.179205 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.179084    4559 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 22:14:21.179205 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.179117    4559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 22:14:21.179561 ip-10-0-138-2 systemd[1]: Started Kubernetes Kubelet.
Oct 29 22:14:21.186076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.179197    4559 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 22:14:21.186076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.182522    4559 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 22:14:21.186076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.183433    4559 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 22:14:21.186076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.184074    4559 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 22:14:21.186076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.184450    4559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 22:14:21.186076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.184513    4559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:08:42 +0000 UTC" deadline="2028-05-14 16:39:35.358679294 +0000 UTC"
Oct 29 22:14:21.186076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.184562    4559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13506h25m14.174121489s"
Oct 29 22:14:21.186076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.184757    4559 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 22:14:21.186076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.184770    4559 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 22:14:21.186076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.184905    4559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 22:14:21.186076 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:21.185799    4559 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-2.ec2.internal\" not found"
Oct 29 22:14:21.187324 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187288    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="807ed4de-272f-4bb2-bf26-cc33f6967771" volumeName="kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-trusted-ca-bundle" seLinuxMountContext=""
Oct 29 22:14:21.187444 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187339    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc8f498e-4039-43ef-91f7-036179511f01" volumeName="kubernetes.io/projected/cc8f498e-4039-43ef-91f7-036179511f01-kube-api-access-ffhsv" seLinuxMountContext=""
Oct 29 22:14:21.187444 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187354    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4a55136-e7f7-46ac-b58e-24e95ab0aea8" volumeName="kubernetes.io/configmap/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-config-volume" seLinuxMountContext=""
Oct 29 22:14:21.187444 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187374    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d637bbed-2662-4313-9639-e7a3bc148955" volumeName="kubernetes.io/secret/d637bbed-2662-4313-9639-e7a3bc148955-cert" seLinuxMountContext=""
Oct 29 22:14:21.187444 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187398    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37a98266-fad3-440d-95d4-85de9ad6f476" volumeName="kubernetes.io/projected/37a98266-fad3-440d-95d4-85de9ad6f476-kube-api-access-n27db" seLinuxMountContext=""
Oct 29 22:14:21.187444 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187413    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="807ed4de-272f-4bb2-bf26-cc33f6967771" volumeName="kubernetes.io/projected/807ed4de-272f-4bb2-bf26-cc33f6967771-kube-api-access-kq8zc" seLinuxMountContext=""
Oct 29 22:14:21.187444 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187436    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc8f498e-4039-43ef-91f7-036179511f01" volumeName="kubernetes.io/configmap/cc8f498e-4039-43ef-91f7-036179511f01-iptables-alerter-script" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187455    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="198638bc-ff2e-4c81-a599-616e7b015749" volumeName="kubernetes.io/secret/198638bc-ff2e-4c81-a599-616e7b015749-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187476    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23921851-35c8-450c-a184-fe5ed0b1c6a7" volumeName="kubernetes.io/projected/23921851-35c8-450c-a184-fe5ed0b1c6a7-kube-api-access-x2bf4" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187503    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="807ed4de-272f-4bb2-bf26-cc33f6967771" volumeName="kubernetes.io/empty-dir/807ed4de-272f-4bb2-bf26-cc33f6967771-archives-path" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187511    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02990faa-cba3-4132-9ca7-fe0f7202bbe5" volumeName="kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187539    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="198638bc-ff2e-4c81-a599-616e7b015749" volumeName="kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-env-overrides" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187547    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21" volumeName="kubernetes.io/secret/1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21-agent-certs" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187563    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c51b17e1-8c9b-44cb-b0dc-aed1c4136a55" volumeName="kubernetes.io/projected/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-kube-api-access-vsljd" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187573    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4a55136-e7f7-46ac-b58e-24e95ab0aea8" volumeName="kubernetes.io/projected/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-kube-api-access-5blnb" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187617    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3c107fc-072e-45ce-b4cd-acbe1284be88" volumeName="kubernetes.io/empty-dir/f3c107fc-072e-45ce-b4cd-acbe1284be88-tmp" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187631    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3c107fc-072e-45ce-b4cd-acbe1284be88" volumeName="kubernetes.io/empty-dir/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-tuned" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187649    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3c107fc-072e-45ce-b4cd-acbe1284be88" volumeName="kubernetes.io/projected/f3c107fc-072e-45ce-b4cd-acbe1284be88-kube-api-access-m8cpn" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187659    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21" volumeName="kubernetes.io/configmap/1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21-konnectivity-ca" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187679    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23921851-35c8-450c-a184-fe5ed0b1c6a7" volumeName="kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187689    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37a98266-fad3-440d-95d4-85de9ad6f476" volumeName="kubernetes.io/configmap/37a98266-fad3-440d-95d4-85de9ad6f476-cni-binary-copy" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187696    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="807ed4de-272f-4bb2-bf26-cc33f6967771" volumeName="kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-service-ca-bundle" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187704    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23921851-35c8-450c-a184-fe5ed0b1c6a7" volumeName="kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-cni-binary-copy" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187719    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3104d9ac-59f1-406b-9d63-7af7892b40bf" volumeName="kubernetes.io/secret/3104d9ac-59f1-406b-9d63-7af7892b40bf-signing-key" seLinuxMountContext=""
Oct 29 22:14:21.187722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187732    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f18ecc2-6ee2-47fa-befa-83065346a118" volumeName="kubernetes.io/empty-dir/8f18ecc2-6ee2-47fa-befa-83065346a118-data-volume" seLinuxMountContext=""
Oct 29 22:14:21.188279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187752    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b3ec29f-f4e7-4811-83ab-50680e0e60e7" volumeName="kubernetes.io/projected/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-kube-api-access-xsrvw" seLinuxMountContext=""
Oct 29 22:14:21.188279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187764    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a66f6cc5-c6c7-476c-96e9-cd1fb85a78de" volumeName="kubernetes.io/projected/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-kube-api-access-pbch6" seLinuxMountContext=""
Oct 29 22:14:21.188279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187775    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="198638bc-ff2e-4c81-a599-616e7b015749" volumeName="kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-ovnkube-script-lib" seLinuxMountContext=""
Oct 29 22:14:21.188279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187803    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="198638bc-ff2e-4c81-a599-616e7b015749" volumeName="kubernetes.io/projected/198638bc-ff2e-4c81-a599-616e7b015749-kube-api-access-ml2q5" seLinuxMountContext=""
Oct 29 22:14:21.188279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187842    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23921851-35c8-450c-a184-fe5ed0b1c6a7" volumeName="kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 29 22:14:21.188279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187854    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="486bbe97-6300-49e9-a0f3-c9d316333048" volumeName="kubernetes.io/projected/486bbe97-6300-49e9-a0f3-c9d316333048-kube-api-access-lxlcz" seLinuxMountContext=""
Oct 29 22:14:21.188279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187878    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c03b55ec-8ed6-42d6-895c-2e7213a2ba39" volumeName="kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w" seLinuxMountContext=""
Oct 29 22:14:21.188279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187894    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c51b17e1-8c9b-44cb-b0dc-aed1c4136a55" volumeName="kubernetes.io/configmap/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-serviceca" seLinuxMountContext=""
Oct 29 22:14:21.188279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187915    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4a55136-e7f7-46ac-b58e-24e95ab0aea8" volumeName="kubernetes.io/empty-dir/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-tmp-dir" seLinuxMountContext=""
Oct 29 22:14:21.188279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187932    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="198638bc-ff2e-4c81-a599-616e7b015749" volumeName="kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-ovnkube-config" seLinuxMountContext=""
Oct 29 22:14:21.188279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.187966    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37a98266-fad3-440d-95d4-85de9ad6f476" volumeName="kubernetes.io/configmap/37a98266-fad3-440d-95d4-85de9ad6f476-multus-daemon-config" seLinuxMountContext=""
Oct 29 22:14:21.188279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.188014    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f18ecc2-6ee2-47fa-befa-83065346a118" volumeName="kubernetes.io/projected/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-api-access-qclfj" seLinuxMountContext=""
Oct 29 22:14:21.188279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.188036    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a66f6cc5-c6c7-476c-96e9-cd1fb85a78de" volumeName="kubernetes.io/empty-dir/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-tmp-dir" seLinuxMountContext=""
Oct 29 22:14:21.188279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.188048    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d637bbed-2662-4313-9639-e7a3bc148955" volumeName="kubernetes.io/projected/d637bbed-2662-4313-9639-e7a3bc148955-kube-api-access-zm7gg" seLinuxMountContext=""
Oct 29 22:14:21.188279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.188080    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5cf9217-fa53-45ec-b7fa-a80520684473" volumeName="kubernetes.io/projected/f5cf9217-fa53-45ec-b7fa-a80520684473-kube-api-access-gbxms" seLinuxMountContext=""
Oct 29 22:14:21.188685 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.188314    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3104d9ac-59f1-406b-9d63-7af7892b40bf" volumeName="kubernetes.io/configmap/3104d9ac-59f1-406b-9d63-7af7892b40bf-signing-cabundle" seLinuxMountContext=""
Oct 29 22:14:21.188685 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.188337    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3104d9ac-59f1-406b-9d63-7af7892b40bf" volumeName="kubernetes.io/projected/3104d9ac-59f1-406b-9d63-7af7892b40bf-kube-api-access-5qv6v" seLinuxMountContext=""
Oct 29 22:14:21.188685 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.188350    4559 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 22:14:21.188685 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.188358    4559 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 22:14:21.189027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.189008    4559 factory.go:55] Registering systemd factory
Oct 29 22:14:21.189027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.189029    4559 factory.go:223] Registration of the systemd container factory successfully
Oct 29 22:14:21.189335 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.189284    4559 factory.go:153] Registering CRI-O factory
Oct 29 22:14:21.189335 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.189296    4559 factory.go:223] Registration of the crio container factory successfully
Oct 29 22:14:21.189429 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.189358    4559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 22:14:21.189429 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.189401    4559 factory.go:103] Registering Raw factory
Oct 29 22:14:21.189429 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.189418    4559 manager.go:1199] Started watching for new ooms in manager
Oct 29 22:14:21.191838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.190048    4559 manager.go:319] Starting recovery of all containers
Oct 29 22:14:21.209767 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.209743    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:14:21.213062 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.213040    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:14:21.217729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.217698    4559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 22:14:21.219198 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.219175    4559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 22:14:21.219198 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.219200    4559 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 22:14:21.219405 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.219224    4559 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 22:14:21.219405 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:21.219285    4559 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 29 22:14:21.231445 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.231416    4559 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:14:21.231664 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.231645    4559 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 29 22:14:21.242671 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.242641    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:14:21.243117 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.243075    4559 generic.go:356] "Generic (PLEG): container finished" podID="198638bc-ff2e-4c81-a599-616e7b015749" containerID="ce9ec1ef6833ca17735f0151ac7ac8762b21a8bbe3505e4fcae77900fa421af2" exitCode=1
Oct 29 22:14:21.244453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.244427    4559 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:14:21.244921 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.244899    4559 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:14:21.248473 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.248452    4559 generic.go:356] "Generic (PLEG): container finished" podID="23921851-35c8-450c-a184-fe5ed0b1c6a7" containerID="67e7f9ec686032ba4ec1745c0ceeb139e6b123dd6d97d256b9853f882083ccb4" exitCode=0
Oct 29 22:14:21.248473 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.248471    4559 generic.go:356] "Generic (PLEG): container finished" podID="23921851-35c8-450c-a184-fe5ed0b1c6a7" containerID="cab18108f05777d958c4d1fd9caa34044465677ca33f285a2a6404ebc0033933" exitCode=0
Oct 29 22:14:21.248575 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.248477    4559 generic.go:356] "Generic (PLEG): container finished" podID="23921851-35c8-450c-a184-fe5ed0b1c6a7" containerID="c478d1186d378edf7897599cca23cc28e9e3c65f67e35ff317683462457c52b3" exitCode=0
Oct 29 22:14:21.248575 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.248483    4559 generic.go:356] "Generic (PLEG): container finished" podID="23921851-35c8-450c-a184-fe5ed0b1c6a7" containerID="8325e667285f3c571437d602d542e16d92455faec8a989b223974a61ae5c604c" exitCode=0
Oct 29 22:14:21.248575 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.248488    4559 generic.go:356] "Generic (PLEG): container finished" podID="23921851-35c8-450c-a184-fe5ed0b1c6a7" containerID="8adcd58764927d9b5d90059d1fa7dcf695570f568f363f21120a96d88938d8c7" exitCode=0
Oct 29 22:14:21.248575 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.248493    4559 generic.go:356] "Generic (PLEG): container finished" podID="23921851-35c8-450c-a184-fe5ed0b1c6a7" containerID="cafd1faddf108ece9a07c7e7cc263b6b4f221f6bfd8515eea57a9543e437620c" exitCode=0
Oct 29 22:14:21.267318 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.267229    4559 generic.go:356] "Generic (PLEG): container finished" podID="4504dba633558f5dac66e017907a0e24" containerID="d3ff5a021aa4cf406ec0b427aaeb0a7addf126dd687ebb60f3e1d612a4098d93" exitCode=0
Oct 29 22:14:21.274159 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.274138    4559 manager.go:324] Recovery completed
Oct 29 22:14:21.286032 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:21.286007    4559 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-2.ec2.internal\" not found"
Oct 29 22:14:21.287578 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.287561    4559 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:14:21.288752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.288734    4559 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:14:21.288840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.288769    4559 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:14:21.288840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.288783    4559 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:14:21.290363 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.290345    4559 cpu_manager.go:226] "Starting" policy="none"
Oct 29 22:14:21.290363 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.290360    4559 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 22:14:21.290487 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.290380    4559 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 22:14:21.290487 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.290349    4559 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:14:21.290558 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.290525    4559 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 29 22:14:21.290558 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.290541    4559 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 29 22:14:21.290620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.290563    4559 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 29 22:14:21.290620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.290571    4559 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 29 22:14:21.290620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.290580    4559 policy_none.go:50] "Start"
Oct 29 22:14:21.290620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.290601    4559 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 22:14:21.290620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.290612    4559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 22:14:21.291256 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.291240    4559 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:14:21.291329 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.291273    4559 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:14:21.291329 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.291288    4559 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:14:21.295822 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.295787    4559 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 29 22:14:21.295822 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.295819    4559 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 29 22:14:21.295822 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.295826    4559 policy_none.go:44] "Start"
Oct 29 22:14:21.298749 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.298733    4559 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:14:21.300150 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.300123    4559 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 22:14:21.305358 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.305340    4559 manager.go:341] "Starting Device Plugin manager"
Oct 29 22:14:21.305452 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:21.305369    4559 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 22:14:21.305452 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.305378    4559 server.go:87] "Starting device plugin registration server"
Oct 29 22:14:21.305715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.305700    4559 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 22:14:21.305778 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.305718    4559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 22:14:21.305778 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.305772    4559 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 22:14:21.305949 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.305903    4559 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 22:14:21.305949 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.305946    4559 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 22:14:21.306053 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.305954    4559 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 22:14:21.306102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.305991    4559 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:14:21.305935204Z","UUID":"36b0790d-80ab-41e3-a495-87b3a9d01784","Handler":null,"Name":"","Endpoint":""}
Oct 29 22:14:21.307359 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:21.307332    4559 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 22:14:21.307572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.307552    4559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 22:14:21.307632 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.307579    4559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 22:14:21.319379 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.319329    4559 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"]
Oct 29 22:14:21.319915 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.319884    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal" event={"ID":"c506ffe79e14398e87cdea51ca1d1f0f","Type":"ContainerStarted","Data":"5a15770ec5bac91693e0518d9cb121c031c8d3b3a9c3932c948f968bac048daf"}
Oct 29 22:14:21.319978 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.319923    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal" event={"ID":"c506ffe79e14398e87cdea51ca1d1f0f","Type":"ContainerStarted","Data":"09ec796e175c68a7ecf208e3fd25136253c240fb603980b9d7cf51f6c295cad2"}
Oct 29 22:14:21.319978 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.319946    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal" event={"ID":"4504dba633558f5dac66e017907a0e24","Type":"ContainerStarted","Data":"0d9ce6c8ecd4c9cd0cfd9fb5192906b08ab1d0f0b16cf4351da47d9d8c78f9d3"}
Oct 29 22:14:21.319978 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.319958    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal" event={"ID":"4504dba633558f5dac66e017907a0e24","Type":"ContainerDied","Data":"d3ff5a021aa4cf406ec0b427aaeb0a7addf126dd687ebb60f3e1d612a4098d93"}
Oct 29 22:14:21.320092 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.319984    4559 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.320092 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.319986    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal" event={"ID":"4504dba633558f5dac66e017907a0e24","Type":"ContainerStarted","Data":"e075d51ca14e4f7237ba867d6845ddcd67c963a12ee720c3c8786c47ebcebac9"}
Oct 29 22:14:21.320168 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.320107    4559 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.350683 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.350657    4559 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:14:21.350839 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:21.350703    4559 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-2.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.353392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.353378    4559 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:14:21.353438 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:21.353414    4559 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.386047 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.386008    4559 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.406852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.406829    4559 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:14:21.407342 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.407326    4559 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:14:21.407405 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:21.407373    4559 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.407405 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.407386    4559 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.407946 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.407926    4559 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:14:21.408044 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.407961    4559 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:14:21.408044 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.407972    4559 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-2.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:14:21.408044 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.408014    4559 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.430287 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.430259    4559 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:14:21.430428 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:21.430308    4559 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-2.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.432491 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.432475    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4504dba633558f5dac66e017907a0e24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"4504dba633558f5dac66e017907a0e24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.432549 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.432498    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4504dba633558f5dac66e017907a0e24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"4504dba633558f5dac66e017907a0e24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.432549 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.432516    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c506ffe79e14398e87cdea51ca1d1f0f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-2.ec2.internal\" (UID: \"c506ffe79e14398e87cdea51ca1d1f0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.447510 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.445581    4559 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.447510 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.445659    4559 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.533542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.533470    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4504dba633558f5dac66e017907a0e24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"4504dba633558f5dac66e017907a0e24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.533542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.533505    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4504dba633558f5dac66e017907a0e24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"4504dba633558f5dac66e017907a0e24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.533542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.533527    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c506ffe79e14398e87cdea51ca1d1f0f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-2.ec2.internal\" (UID: \"c506ffe79e14398e87cdea51ca1d1f0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.533742 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.533579    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c506ffe79e14398e87cdea51ca1d1f0f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-2.ec2.internal\" (UID: \"c506ffe79e14398e87cdea51ca1d1f0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.533742 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.533580    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4504dba633558f5dac66e017907a0e24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"4504dba633558f5dac66e017907a0e24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:21.533742 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:21.533618    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4504dba633558f5dac66e017907a0e24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" (UID: \"4504dba633558f5dac66e017907a0e24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:22.171957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.171925    4559 apiserver.go:52] "Watching apiserver"
Oct 29 22:14:22.226451 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.226416    4559 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 22:14:22.226955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.226929    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-v552v","openshift-insights/periodic-gathering-hc4lm-rqsvk","kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw","openshift-cluster-node-tuning-operator/tuned-pvswk","openshift-dns/dns-default-xp49w","openshift-service-ca/service-ca-5fffc97bdf-x4nbg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal","openshift-ingress-canary/ingress-canary-fzpgj","openshift-network-operator/iptables-alerter-xfm9m","openshift-multus/multus-7q7dr","openshift-multus/network-metrics-daemon-t48mn","openshift-insights/insights-runtime-extractor-dnx6b","openshift-image-registry/node-ca-8bl7p","kube-system/global-pull-secret-syncer-p5bdf","openshift-multus/multus-additional-cni-plugins-bn9z2","openshift-network-diagnostics/network-check-target-5pq4b","openshift-ovn-kubernetes/ovnkube-node-zlnh6","openshift-kube-storage-version-migrator/migrator-66df5d496b-5774f","openshift-dns/node-resolver-jls5s"]
Oct 29 22:14:22.230496 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.230476    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:14:22.231947 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.231931    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:22.232924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.232904    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:22.244093 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.244062    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p5dkc\""
Oct 29 22:14:22.245048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.244231    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-cdnlg\""
Oct 29 22:14:22.245048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.244315    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 22:14:22.245048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.244448    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 22:14:22.245048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.244536    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 22:14:22.245048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.244643    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8zjwp\""
Oct 29 22:14:22.245048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.244743    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fz5pc\""
Oct 29 22:14:22.245048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.244864    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 22:14:22.245048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.244876    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 22:14:22.245048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.244940    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 22:14:22.245048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.244983    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 22:14:22.245048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.244992    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 22:14:22.245582 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.245160    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 22:14:22.245582 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.245291    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 22:14:22.245582 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.245405    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 22:14:22.245705 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.245654    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 22:14:22.246282 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.245967    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 22:14:22.253358 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.253337    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 22:14:22.253461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.253376    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 22:14:22.253461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.253410    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 22:14:22.253461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.253430    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 22:14:22.254068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.254051    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 22:14:22.254120 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.254071    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-977wj\""
Oct 29 22:14:22.254209 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.254196    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2nqlz\""
Oct 29 22:14:22.254256 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.254208    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7vbxc\""
Oct 29 22:14:22.254256 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.254247    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bjh7c\""
Oct 29 22:14:22.254341 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.254290    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bwm9d\""
Oct 29 22:14:22.254341 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.254306    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mwjl8\""
Oct 29 22:14:22.254433 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.254403    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f584j\""
Oct 29 22:14:22.254433 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.254406    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kr68l\""
Oct 29 22:14:22.254510 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.254461    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-v7npq\""
Oct 29 22:14:22.255915 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.255900    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 22:14:22.256001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.255948    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 22:14:22.256001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.255984    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 22:14:22.256099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256014    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 22:14:22.256099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256084    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 22:14:22.256274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256123    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hkdgm\""
Oct 29 22:14:22.256274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256127    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 22:14:22.256274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256141    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 22:14:22.256274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256160    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 22:14:22.256274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256168    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 22:14:22.256274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256172    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xsn6t\""
Oct 29 22:14:22.256274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256178    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:22.256274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256175    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 22:14:22.256274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256192    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 22:14:22.256670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256354    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 22:14:22.256670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256454    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 22:14:22.256670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256481    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xpwxr\""
Oct 29 22:14:22.256670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256554    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 22:14:22.256670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256583    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:22.256670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256601    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 29 22:14:22.256670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256613    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 22:14:22.256670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256601    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 22:14:22.256670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256660    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kj4hx\""
Oct 29 22:14:22.257042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256697    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 29 22:14:22.257042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256757    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 29 22:14:22.257042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256765    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 22:14:22.257042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256929    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:22.257042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256975    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 29 22:14:22.257042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256988    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 29 22:14:22.257042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.256999    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 22:14:22.257360 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.257347    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 29 22:14:22.257360 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.257355    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 22:14:22.257412 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.257400    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 22:14:22.257461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.257450    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hl5hv\""
Oct 29 22:14:22.267539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.267518    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 29 22:14:22.269772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.269757    4559 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:22.269887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.269875    4559 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:22.286252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.286228    4559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 22:14:22.293077 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.293042    4559 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:14:22.293183 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:22.293126    4559 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-2.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:22.293825 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.293784    4559 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:14:22.293913 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:22.293850    4559 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-2.ec2.internal"
Oct 29 22:14:22.337959 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.337923    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-device-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.338107 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.337961    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-cnibin\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.338107 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.337987    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-socket-dir-parent\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.338107 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338042    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:14:22.338107 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338085    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc8f498e-4039-43ef-91f7-036179511f01-host-slash\") pod \"iptables-alerter-xfm9m\" (UID: \"cc8f498e-4039-43ef-91f7-036179511f01\") " pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:14:22.338250 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338118    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/198638bc-ff2e-4c81-a599-616e7b015749-ovn-node-metrics-cert\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.338250 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338180    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-socket-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.338250 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338208    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-k8s-cni-cncf-io\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.338250 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338226    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-kubelet\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.338421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338270    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21-agent-certs\") pod \"konnectivity-agent-v552v\" (UID: \"1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21\") " pod="kube-system/konnectivity-agent-v552v"
Oct 29 22:14:22.338421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338297    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.338421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338328    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/198638bc-ff2e-4c81-a599-616e7b015749-ovn-node-metrics-cert\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.338421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338341    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-cni-netd\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.338421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338378    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f18ecc2-6ee2-47fa-befa-83065346a118-data-volume\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:22.338594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338435    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-cni-multus\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.338594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338484    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f18ecc2-6ee2-47fa-befa-83065346a118-data-volume\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:22.338594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338485    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-etc-kubernetes\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.338594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338529    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:14:22.338594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338554    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2bf4\" (UniqueName: \"kubernetes.io/projected/23921851-35c8-450c-a184-fe5ed0b1c6a7-kube-api-access-x2bf4\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:14:22.338594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338574    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-sys-fs\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.338870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338637    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21-agent-certs\") pod \"konnectivity-agent-v552v\" (UID: \"1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21\") " pod="kube-system/konnectivity-agent-v552v"
Oct 29 22:14:22.338870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338745    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d637bbed-2662-4313-9639-e7a3bc148955-cert\") pod \"ingress-canary-fzpgj\" (UID: \"d637bbed-2662-4313-9639-e7a3bc148955\") " pod="openshift-ingress-canary/ingress-canary-fzpgj"
Oct 29 22:14:22.338870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338765    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:22.338870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338782    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-tmp-dir\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:22.338870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338797    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-modprobe-d\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.339087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338909    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:14:22.339087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338935    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:22.339087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338946    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-tmp-dir\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:22.339087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.338986    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-run-netns\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.339087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339006    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbch6\" (UniqueName: \"kubernetes.io/projected/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-kube-api-access-pbch6\") pod \"node-resolver-jls5s\" (UID: \"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de\") " pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:14:22.339087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339008    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d637bbed-2662-4313-9639-e7a3bc148955-cert\") pod \"ingress-canary-fzpgj\" (UID: \"d637bbed-2662-4313-9639-e7a3bc148955\") " pod="openshift-ingress-canary/ingress-canary-fzpgj"
Oct 29 22:14:22.339087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339021    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlcz\" (UniqueName: \"kubernetes.io/projected/486bbe97-6300-49e9-a0f3-c9d316333048-kube-api-access-lxlcz\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.339087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339037    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/807ed4de-272f-4bb2-bf26-cc33f6967771-archives-path\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:22.339087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339069    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5blnb\" (UniqueName: \"kubernetes.io/projected/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-kube-api-access-5blnb\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:22.339476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339094    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-kubernetes\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.339476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339110    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-multus-certs\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.339476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339127    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:14:22.339476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339226    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02990faa-cba3-4132-9ca7-fe0f7202bbe5-kubelet-config\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:22.339476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339303    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/807ed4de-272f-4bb2-bf26-cc33f6967771-archives-path\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:22.339476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339340    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-slash\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.339476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339340    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:14:22.339476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339377    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-log-socket\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.339476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339412    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-registration-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.339476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339436    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-run\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.339476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339453    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n27db\" (UniqueName: \"kubernetes.io/projected/37a98266-fad3-440d-95d4-85de9ad6f476-kube-api-access-n27db\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.339476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339468    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-system-cni-dir\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:14:22.340026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339499    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-cnibin\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:14:22.340026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339542    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-var-lib-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.340026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339580    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-etc-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.340026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339611    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml2q5\" (UniqueName: \"kubernetes.io/projected/198638bc-ff2e-4c81-a599-616e7b015749-kube-api-access-ml2q5\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.340026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339639    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-etc-selinux\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.340026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339733    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7gg\" (UniqueName: \"kubernetes.io/projected/d637bbed-2662-4313-9639-e7a3bc148955-kube-api-access-zm7gg\") pod \"ingress-canary-fzpgj\" (UID: \"d637bbed-2662-4313-9639-e7a3bc148955\") " pod="openshift-ingress-canary/ingress-canary-fzpgj"
Oct 29 22:14:22.340026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339841    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-kubelet\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.340026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339863    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-env-overrides\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.340026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339938    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f18ecc2-6ee2-47fa-befa-83065346a118-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:22.340026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339981    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysctl-d\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.340026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.339999    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-env-overrides\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.340026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340014    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-host\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.340520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340053    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-system-cni-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.340520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340073    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37a98266-fad3-440d-95d4-85de9ad6f476-cni-binary-copy\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.340520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340088    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-cni-bin\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.340520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340104    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsljd\" (UniqueName: \"kubernetes.io/projected/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-kube-api-access-vsljd\") pod \"node-ca-8bl7p\" (UID: \"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55\") " pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:14:22.340520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340134    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxms\" (UniqueName: \"kubernetes.io/projected/f5cf9217-fa53-45ec-b7fa-a80520684473-kube-api-access-gbxms\") pod \"migrator-66df5d496b-5774f\" (UID: \"f5cf9217-fa53-45ec-b7fa-a80520684473\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5774f"
Oct 29 22:14:22.340520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340154    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysctl-conf\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.340520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340218    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-cni-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.340520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340268    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-conf-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.340520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340298    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21-konnectivity-ca\") pod \"konnectivity-agent-v552v\" (UID: \"1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21\") " pod="kube-system/konnectivity-agent-v552v"
Oct 29 22:14:22.340520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340376    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:14:22.340520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340390    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37a98266-fad3-440d-95d4-85de9ad6f476-cni-binary-copy\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.340520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340411    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-cni-bin\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.340520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340479    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:14:22.340520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340521    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23921851-35c8-450c-a184-fe5ed0b1c6a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:14:22.340520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340523    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21-konnectivity-ca\") pod \"konnectivity-agent-v552v\" (UID: \"1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21\") " pod="kube-system/konnectivity-agent-v552v"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340524    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq8zc\" (UniqueName: \"kubernetes.io/projected/807ed4de-272f-4bb2-bf26-cc33f6967771-kube-api-access-kq8zc\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340568    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysconfig\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340592    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-var-lib-kubelet\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340626    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340662    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-serviceca\") pod \"node-ca-8bl7p\" (UID: \"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55\") " pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340688    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-hosts-file\") pod \"node-resolver-jls5s\" (UID: \"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de\") " pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340709    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-trusted-ca-bundle\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340727    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvr8w\" (UniqueName: \"kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w\") pod \"network-check-target-5pq4b\" (UID: \"c03b55ec-8ed6-42d6-895c-2e7213a2ba39\") " pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340749    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-hostroot\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340775    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffhsv\" (UniqueName: \"kubernetes.io/projected/cc8f498e-4039-43ef-91f7-036179511f01-kube-api-access-ffhsv\") pod \"iptables-alerter-xfm9m\" (UID: \"cc8f498e-4039-43ef-91f7-036179511f01\") " pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340796    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-node-log\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340840    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-host\") pod \"node-ca-8bl7p\" (UID: \"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55\") " pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340866    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qv6v\" (UniqueName: \"kubernetes.io/projected/3104d9ac-59f1-406b-9d63-7af7892b40bf-kube-api-access-5qv6v\") pod \"service-ca-5fffc97bdf-x4nbg\" (UID: \"3104d9ac-59f1-406b-9d63-7af7892b40bf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340890    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02990faa-cba3-4132-9ca7-fe0f7202bbe5-dbus\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340905    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-serviceca\") pod \"node-ca-8bl7p\" (UID: \"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55\") " pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340921    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f18ecc2-6ee2-47fa-befa-83065346a118-crio-socket\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.340969    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-lib-modules\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341037    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3c107fc-072e-45ce-b4cd-acbe1284be88-tmp\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341091    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37a98266-fad3-440d-95d4-85de9ad6f476-multus-daemon-config\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341115    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsrvw\" (UniqueName: \"kubernetes.io/projected/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-kube-api-access-xsrvw\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341133    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3104d9ac-59f1-406b-9d63-7af7892b40bf-signing-cabundle\") pod \"service-ca-5fffc97bdf-x4nbg\" (UID: \"3104d9ac-59f1-406b-9d63-7af7892b40bf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341149    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-systemd\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341172    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-sys\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341147    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3c107fc-072e-45ce-b4cd-acbe1284be88-tmp\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341198    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-tuned\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.341254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341258    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-config-volume\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341276    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-tuned\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341305    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-systemd-units\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341349    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-systemd\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341377    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-tmp-dir\") pod \"node-resolver-jls5s\" (UID: \"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de\") " pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341424    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341481    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-tmp-dir\") pod \"node-resolver-jls5s\" (UID: \"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de\") " pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341520    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-service-ca-bundle\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341556    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-netns\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341578    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc8f498e-4039-43ef-91f7-036179511f01-iptables-alerter-script\") pod \"iptables-alerter-xfm9m\" (UID: \"cc8f498e-4039-43ef-91f7-036179511f01\") " pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341620    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-ovnkube-config\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341655    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-trusted-ca-bundle\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341715    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-ovnkube-script-lib\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341746    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3104d9ac-59f1-406b-9d63-7af7892b40bf-signing-key\") pod \"service-ca-5fffc97bdf-x4nbg\" (UID: \"3104d9ac-59f1-406b-9d63-7af7892b40bf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341843    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341871    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qclfj\" (UniqueName: \"kubernetes.io/projected/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-api-access-qclfj\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341903    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3104d9ac-59f1-406b-9d63-7af7892b40bf-signing-cabundle\") pod \"service-ca-5fffc97bdf-x4nbg\" (UID: \"3104d9ac-59f1-406b-9d63-7af7892b40bf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341916    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8cpn\" (UniqueName: \"kubernetes.io/projected/f3c107fc-072e-45ce-b4cd-acbe1284be88-kube-api-access-m8cpn\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341965    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-ovn\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.341983    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-ovnkube-script-lib\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.342015    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3104d9ac-59f1-406b-9d63-7af7892b40bf-signing-key\") pod \"service-ca-5fffc97bdf-x4nbg\" (UID: \"3104d9ac-59f1-406b-9d63-7af7892b40bf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.342017    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc8f498e-4039-43ef-91f7-036179511f01-iptables-alerter-script\") pod \"iptables-alerter-xfm9m\" (UID: \"cc8f498e-4039-43ef-91f7-036179511f01\") " pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.342044    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37a98266-fad3-440d-95d4-85de9ad6f476-multus-daemon-config\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.342080    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-run-ovn-kubernetes\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.342105    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02990faa-cba3-4132-9ca7-fe0f7202bbe5-original-pull-secret\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.342236    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/198638bc-ff2e-4c81-a599-616e7b015749-ovnkube-config\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.342267    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-config-volume\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:22.342381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.342364    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-service-ca-bundle\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:22.356749 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.356722    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2bf4\" (UniqueName: \"kubernetes.io/projected/23921851-35c8-450c-a184-fe5ed0b1c6a7-kube-api-access-x2bf4\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:14:22.357159 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.357131    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlcz\" (UniqueName: \"kubernetes.io/projected/486bbe97-6300-49e9-a0f3-c9d316333048-kube-api-access-lxlcz\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.357245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.357162    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbch6\" (UniqueName: \"kubernetes.io/projected/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-kube-api-access-pbch6\") pod \"node-resolver-jls5s\" (UID: \"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de\") " pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:14:22.357245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.357235    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5blnb\" (UniqueName: \"kubernetes.io/projected/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-kube-api-access-5blnb\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:22.357403 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.357382    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n27db\" (UniqueName: \"kubernetes.io/projected/37a98266-fad3-440d-95d4-85de9ad6f476-kube-api-access-n27db\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.357701 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.357682    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsljd\" (UniqueName: \"kubernetes.io/projected/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-kube-api-access-vsljd\") pod \"node-ca-8bl7p\" (UID: \"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55\") " pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:14:22.357760 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.357724    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7gg\" (UniqueName: \"kubernetes.io/projected/d637bbed-2662-4313-9639-e7a3bc148955-kube-api-access-zm7gg\") pod \"ingress-canary-fzpgj\" (UID: \"d637bbed-2662-4313-9639-e7a3bc148955\") " pod="openshift-ingress-canary/ingress-canary-fzpgj"
Oct 29 22:14:22.358159 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.358136    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml2q5\" (UniqueName: \"kubernetes.io/projected/198638bc-ff2e-4c81-a599-616e7b015749-kube-api-access-ml2q5\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.359636 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.359619    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvr8w\" (UniqueName: \"kubernetes.io/projected/c03b55ec-8ed6-42d6-895c-2e7213a2ba39-kube-api-access-bvr8w\") pod \"network-check-target-5pq4b\" (UID: \"c03b55ec-8ed6-42d6-895c-2e7213a2ba39\") " pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:14:22.359999 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.359980    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxms\" (UniqueName: \"kubernetes.io/projected/f5cf9217-fa53-45ec-b7fa-a80520684473-kube-api-access-gbxms\") pod \"migrator-66df5d496b-5774f\" (UID: \"f5cf9217-fa53-45ec-b7fa-a80520684473\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5774f"
Oct 29 22:14:22.360316 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.360291    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffhsv\" (UniqueName: \"kubernetes.io/projected/cc8f498e-4039-43ef-91f7-036179511f01-kube-api-access-ffhsv\") pod \"iptables-alerter-xfm9m\" (UID: \"cc8f498e-4039-43ef-91f7-036179511f01\") " pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:14:22.360389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.360304    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsrvw\" (UniqueName: \"kubernetes.io/projected/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-kube-api-access-xsrvw\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:14:22.360634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.360610    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qclfj\" (UniqueName: \"kubernetes.io/projected/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-api-access-qclfj\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:22.360773 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.360743    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq8zc\" (UniqueName: \"kubernetes.io/projected/807ed4de-272f-4bb2-bf26-cc33f6967771-kube-api-access-kq8zc\") pod \"periodic-gathering-hc4lm-rqsvk\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") " pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:14:22.360849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.360789    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qv6v\" (UniqueName: \"kubernetes.io/projected/3104d9ac-59f1-406b-9d63-7af7892b40bf-kube-api-access-5qv6v\") pod \"service-ca-5fffc97bdf-x4nbg\" (UID: \"3104d9ac-59f1-406b-9d63-7af7892b40bf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x4nbg"
Oct 29 22:14:22.362855 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.362832    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8cpn\" (UniqueName: \"kubernetes.io/projected/f3c107fc-072e-45ce-b4cd-acbe1284be88-kube-api-access-m8cpn\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.442936 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.442863    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:22.442936 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.442893    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-run-netns\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.442936 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.442911    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-kubernetes\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.442936 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.442934    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-multus-certs\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.443183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.442958    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02990faa-cba3-4132-9ca7-fe0f7202bbe5-kubelet-config\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:22.443183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.442978    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-slash\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.443183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.442985    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-run-netns\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.443183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.442997    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-log-socket\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.443183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443013    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-kubernetes\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.443183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443030    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-registration-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.443183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443013    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-multus-certs\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.443183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443033    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-log-socket\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.443183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443063    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02990faa-cba3-4132-9ca7-fe0f7202bbe5-kubelet-config\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:22.443183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443071    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-registration-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.443183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443079    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-slash\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.443183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443093    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-run\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.443183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443128    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-system-cni-dir\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:14:22.443183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443152    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-cnibin\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:14:22.443183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443156    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-run\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.443183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443178    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-system-cni-dir\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:14:22.443183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443185    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-cnibin\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443179    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-var-lib-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443210    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-var-lib-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443225    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-etc-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443247    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-etc-selinux\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443270    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-kubelet\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443280    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-etc-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443308    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-kubelet\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443320    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f18ecc2-6ee2-47fa-befa-83065346a118-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443339    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysctl-d\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443343    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-etc-selinux\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443353    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-host\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443383    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-system-cni-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443405    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-cni-bin\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443422    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysctl-conf\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443435    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-cni-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443451    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-conf-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443468    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f18ecc2-6ee2-47fa-befa-83065346a118-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443473    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-cni-bin\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443505    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysctl-d\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443516    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-conf-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443520    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443490    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-system-cni-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443527    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-host\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443540    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-cni-bin\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443509    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-cni-bin\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443551    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysconfig\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443578    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysctl-conf\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:22.443610    4559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443624    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-var-lib-kubelet\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443648    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-var-lib-kubelet\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443702    4559 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443627    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-sysconfig\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.443852 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:22.443850    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs podName:9b3ec29f-f4e7-4811-83ab-50680e0e60e7 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:22.943795657 +0000 UTC m=+1.855239194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs") pod "network-metrics-daemon-t48mn" (UID: "9b3ec29f-f4e7-4811-83ab-50680e0e60e7") : secret "metrics-daemon-secret" not found
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443705    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444218    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-hosts-file\") pod \"node-resolver-jls5s\" (UID: \"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de\") " pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444305    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-hostroot\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444369    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-node-log\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444431    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-host\") pod \"node-ca-8bl7p\" (UID: \"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55\") " pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444439    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c51b17e1-8c9b-44cb-b0dc-aed1c4136a55-host\") pod \"node-ca-8bl7p\" (UID: \"c51b17e1-8c9b-44cb-b0dc-aed1c4136a55\") " pod="openshift-image-registry/node-ca-8bl7p"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444493    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a66f6cc5-c6c7-476c-96e9-cd1fb85a78de-hosts-file\") pod \"node-resolver-jls5s\" (UID: \"a66f6cc5-c6c7-476c-96e9-cd1fb85a78de\") " pod="openshift-dns/node-resolver-jls5s"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444521    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-hostroot\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444546    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02990faa-cba3-4132-9ca7-fe0f7202bbe5-dbus\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444570    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-node-log\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444578    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f18ecc2-6ee2-47fa-befa-83065346a118-crio-socket\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444625    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-lib-modules\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444654    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-systemd\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444667    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444681    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-sys\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444669    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f18ecc2-6ee2-47fa-befa-83065346a118-crio-socket\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.443713    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-cni-dir\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444714    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-systemd-units\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444733    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-systemd\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444741    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-systemd\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444628    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02990faa-cba3-4132-9ca7-fe0f7202bbe5-dbus\") pod \"global-pull-secret-syncer-p5bdf\" (UID: \"02990faa-cba3-4132-9ca7-fe0f7202bbe5\") " pod="kube-system/global-pull-secret-syncer-p5bdf"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444793    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-sys\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444799    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-systemd\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444803    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-systemd-units\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444861    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444913    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-netns\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444934    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-lib-modules\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444949    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.444952    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-ovn\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445013    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-ovn\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445051    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-netns\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445086    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-run-ovn-kubernetes\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445148    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-device-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.445192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445173    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-cnibin\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445230    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-device-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445264    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-socket-dir-parent\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445295    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445322    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc8f498e-4039-43ef-91f7-036179511f01-host-slash\") pod \"iptables-alerter-xfm9m\" (UID: \"cc8f498e-4039-43ef-91f7-036179511f01\") " pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445412    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23921851-35c8-450c-a184-fe5ed0b1c6a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bn9z2\" (UID: \"23921851-35c8-450c-a184-fe5ed0b1c6a7\") " pod="openshift-multus/multus-additional-cni-plugins-bn9z2"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445446    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-socket-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445476    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-k8s-cni-cncf-io\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445504    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-kubelet\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445570    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445602    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-cni-netd\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445632    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-cni-multus\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445655    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-etc-kubernetes\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445688    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-sys-fs\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445732    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445764    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-modprobe-d\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445917    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f3c107fc-072e-45ce-b4cd-acbe1284be88-etc-modprobe-d\") pod \"tuned-pvswk\" (UID: \"f3c107fc-072e-45ce-b4cd-acbe1284be88\") " pod="openshift-cluster-node-tuning-operator/tuned-pvswk"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.445960    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc8f498e-4039-43ef-91f7-036179511f01-host-slash\") pod \"iptables-alerter-xfm9m\" (UID: \"cc8f498e-4039-43ef-91f7-036179511f01\") " pod="openshift-network-operator/iptables-alerter-xfm9m"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.446008    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-cnibin\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.446047    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-run-ovn-kubernetes\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.446094    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-multus-socket-dir-parent\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.446171    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-socket-dir\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.446634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.446217    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-run-k8s-cni-cncf-io\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.447550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.446779    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f18ecc2-6ee2-47fa-befa-83065346a118-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnx6b\" (UID: \"8f18ecc2-6ee2-47fa-befa-83065346a118\") " pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:22.447550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.446877    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-cni-multus\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.447550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.446928    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-run-openvswitch\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.447550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.446978    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/198638bc-ff2e-4c81-a599-616e7b015749-host-cni-netd\") pod \"ovnkube-node-zlnh6\" (UID: \"198638bc-ff2e-4c81-a599-616e7b015749\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:22.447550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.447045    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/486bbe97-6300-49e9-a0f3-c9d316333048-sys-fs\") pod \"aws-ebs-csi-driver-node-tn7jw\" (UID: \"486bbe97-6300-49e9-a0f3-c9d316333048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tn7jw"
Oct 29 22:14:22.447550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.447100    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-etc-kubernetes\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.447550 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:22.447173    4559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:22.447550 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:22.447317    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls podName:d4a55136-e7f7-46ac-b58e-24e95ab0aea8 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:22.947270469 +0000 UTC m=+1.858714001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls") pod "dns-default-xp49w" (UID: "d4a55136-e7f7-46ac-b58e-24e95ab0aea8") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:22.448497 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.448440    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37a98266-fad3-440d-95d4-85de9ad6f476-host-var-lib-kubelet\") pod \"multus-7q7dr\" (UID: \"37a98266-fad3-440d-95d4-85de9ad6f476\") " pod="openshift-multus/multus-7q7dr"
Oct 29 22:14:22.533477 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.533447    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dnx6b"
Oct 29 22:14:22.858403 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.858319    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dnx6b"]
Oct 29 22:14:22.861385 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:22.861346    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f18ecc2_6ee2_47fa_befa_83065346a118.slice/crio-583842bf3ff4de0e6e32366179651b6957531be47d3e9b3bd6a67d9129c8a348 WatchSource:0}: Error finding container 583842bf3ff4de0e6e32366179651b6957531be47d3e9b3bd6a67d9129c8a348: Status 404 returned error can't find the container with id 583842bf3ff4de0e6e32366179651b6957531be47d3e9b3bd6a67d9129c8a348
Oct 29 22:14:22.872196 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.872168    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8bl7p_c51b17e1-8c9b-44cb-b0dc-aed1c4136a55/node-ca/0.log"
Oct 29 22:14:22.951155 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.951121    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:22.951271 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.951178    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:14:22.951326 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:22.951288    4559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:22.951376 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:22.951365    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls podName:d4a55136-e7f7-46ac-b58e-24e95ab0aea8 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:23.951341656 +0000 UTC m=+2.862785201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls") pod "dns-default-xp49w" (UID: "d4a55136-e7f7-46ac-b58e-24e95ab0aea8") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:22.951434 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:22.951293    4559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 29 22:14:22.951495 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:22.951439    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs podName:9b3ec29f-f4e7-4811-83ab-50680e0e60e7 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:23.951421236 +0000 UTC m=+2.862864781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs") pod "network-metrics-daemon-t48mn" (UID: "9b3ec29f-f4e7-4811-83ab-50680e0e60e7") : secret "metrics-daemon-secret" not found
Oct 29 22:14:22.959773 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:22.959750    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:14:23.275560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:23.275523    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnx6b" event={"ID":"8f18ecc2-6ee2-47fa-befa-83065346a118","Type":"ContainerStarted","Data":"7af2afe160a331b852b1bfd0bb9e50c8cb002bb49aa4a629b40d829fa982825c"}
Oct 29 22:14:23.276093 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:23.275589    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnx6b" event={"ID":"8f18ecc2-6ee2-47fa-befa-83065346a118","Type":"ContainerStarted","Data":"583842bf3ff4de0e6e32366179651b6957531be47d3e9b3bd6a67d9129c8a348"}
Oct 29 22:14:23.472830 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:23.472790    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fzpgj_d637bbed-2662-4313-9639-e7a3bc148955/serve-healthcheck-canary/0.log"
Oct 29 22:14:23.571767 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:23.571692    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:23.957230 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:23.957189    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:14:23.957414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:23.957248    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:23.957414 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:23.957343    4559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:23.957414 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:23.957396    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls podName:d4a55136-e7f7-46ac-b58e-24e95ab0aea8 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:25.957384619 +0000 UTC m=+4.868828167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls") pod "dns-default-xp49w" (UID: "d4a55136-e7f7-46ac-b58e-24e95ab0aea8") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:23.959757 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:23.959736    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b3ec29f-f4e7-4811-83ab-50680e0e60e7-metrics-certs\") pod \"network-metrics-daemon-t48mn\" (UID: \"9b3ec29f-f4e7-4811-83ab-50680e0e60e7\") " pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:14:24.031399 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:24.031363    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t48mn"
Oct 29 22:14:24.276560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:24.276461    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-5774f_f5cf9217-fa53-45ec-b7fa-a80520684473/migrator/0.log"
Oct 29 22:14:24.344038 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:24.344006    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t48mn"]
Oct 29 22:14:24.347006 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:24.346976    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3ec29f_f4e7_4811_83ab_50680e0e60e7.slice/crio-87202e093ec417eb914837190c0250685f840df2110b9f2933d702a121e261bd WatchSource:0}: Error finding container 87202e093ec417eb914837190c0250685f840df2110b9f2933d702a121e261bd: Status 404 returned error can't find the container with id 87202e093ec417eb914837190c0250685f840df2110b9f2933d702a121e261bd
Oct 29 22:14:25.284492 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:25.284458    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnx6b" event={"ID":"8f18ecc2-6ee2-47fa-befa-83065346a118","Type":"ContainerStarted","Data":"61e633920afc38c8d659f187b72cea42182ac2c0bb149764de02e6f1d2d98a78"}
Oct 29 22:14:25.285827 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:25.285785    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t48mn" event={"ID":"9b3ec29f-f4e7-4811-83ab-50680e0e60e7","Type":"ContainerStarted","Data":"87202e093ec417eb914837190c0250685f840df2110b9f2933d702a121e261bd"}
Oct 29 22:14:25.972631 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:25.972587    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:25.975168 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:25.975139    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4a55136-e7f7-46ac-b58e-24e95ab0aea8-metrics-tls\") pod \"dns-default-xp49w\" (UID: \"d4a55136-e7f7-46ac-b58e-24e95ab0aea8\") " pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:26.132568 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:26.132485    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:26.292877 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:26.292844    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t48mn" event={"ID":"9b3ec29f-f4e7-4811-83ab-50680e0e60e7","Type":"ContainerStarted","Data":"de83d5e1e378bd180ed7316aaf88f564b91e44495beeb0435e5c18a15b1df0a7"}
Oct 29 22:14:26.293314 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:26.292912    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t48mn" event={"ID":"9b3ec29f-f4e7-4811-83ab-50680e0e60e7","Type":"ContainerStarted","Data":"7e6a84f5d6945e23eb6960c07d73bc89114be14e3fb758184f46d7bec90383ab"}
Oct 29 22:14:26.294738 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:26.294715    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnx6b" event={"ID":"8f18ecc2-6ee2-47fa-befa-83065346a118","Type":"ContainerStarted","Data":"8d971033fc1e1eeca3cfdec12ecc0de35c138c586aff9ec246c15b2cfe67cee6"}
Oct 29 22:14:26.429489 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:26.429099    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xp49w"]
Oct 29 22:14:26.431844 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:26.431795    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a55136_e7f7_46ac_b58e_24e95ab0aea8.slice/crio-1baab9f5f020e2e7bf7a3773dc99c3597951d46a4e86f3bbbfaca77713a90bfa WatchSource:0}: Error finding container 1baab9f5f020e2e7bf7a3773dc99c3597951d46a4e86f3bbbfaca77713a90bfa: Status 404 returned error can't find the container with id 1baab9f5f020e2e7bf7a3773dc99c3597951d46a4e86f3bbbfaca77713a90bfa
Oct 29 22:14:27.300277 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:27.300219    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xp49w" event={"ID":"d4a55136-e7f7-46ac-b58e-24e95ab0aea8","Type":"ContainerStarted","Data":"1baab9f5f020e2e7bf7a3773dc99c3597951d46a4e86f3bbbfaca77713a90bfa"}
Oct 29 22:14:28.305242 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:28.305150    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xp49w" event={"ID":"d4a55136-e7f7-46ac-b58e-24e95ab0aea8","Type":"ContainerStarted","Data":"8963ff280fcd67154470463e3bf4ec7531fb1b4a018579609a3c4a269ff4e65e"}
Oct 29 22:14:28.305242 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:28.305200    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xp49w" event={"ID":"d4a55136-e7f7-46ac-b58e-24e95ab0aea8","Type":"ContainerStarted","Data":"b96509563bdfe43cf1a1ecf39f4c01b6e982256b529b1dfaa9b88e775cbae41a"}
Oct 29 22:14:28.305654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:28.305305    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:28.517533 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:28.517497    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:14:31.669931 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:31.669893    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:31.670393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:31.670102    4559 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:14:35.724549 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.724512    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-dcb998c6b-xlxth"]
Oct 29 22:14:35.727498 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.727480    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.773498 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.773451    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 29 22:14:35.773498 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.773480    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 29 22:14:35.773757 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.773506    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x9frc\""
Oct 29 22:14:35.773757 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.773560    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 29 22:14:35.773757 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.773622    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 29 22:14:35.774438 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.774416    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dcb998c6b-xlxth"]
Oct 29 22:14:35.838895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.838862    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70e57cd4-a6c4-4299-99bf-a29c21453751-installation-pull-secrets\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.839099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.838900    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70e57cd4-a6c4-4299-99bf-a29c21453751-image-registry-private-configuration\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.839099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.838924    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70e57cd4-a6c4-4299-99bf-a29c21453751-bound-sa-token\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.839099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.839010    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70e57cd4-a6c4-4299-99bf-a29c21453751-trusted-ca\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.839099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.839083    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70e57cd4-a6c4-4299-99bf-a29c21453751-registry-tls\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.839314 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.839120    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70e57cd4-a6c4-4299-99bf-a29c21453751-ca-trust-extracted\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.839314 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.839159    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70e57cd4-a6c4-4299-99bf-a29c21453751-registry-certificates\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.839314 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.839269    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkzfb\" (UniqueName: \"kubernetes.io/projected/70e57cd4-a6c4-4299-99bf-a29c21453751-kube-api-access-zkzfb\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.939598 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.939563    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70e57cd4-a6c4-4299-99bf-a29c21453751-image-registry-private-configuration\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.939598 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.939603    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70e57cd4-a6c4-4299-99bf-a29c21453751-bound-sa-token\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.939844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.939632    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70e57cd4-a6c4-4299-99bf-a29c21453751-trusted-ca\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.939844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.939660    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70e57cd4-a6c4-4299-99bf-a29c21453751-registry-tls\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.939844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.939691    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70e57cd4-a6c4-4299-99bf-a29c21453751-ca-trust-extracted\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.939844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.939721    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70e57cd4-a6c4-4299-99bf-a29c21453751-registry-certificates\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.939844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.939755    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkzfb\" (UniqueName: \"kubernetes.io/projected/70e57cd4-a6c4-4299-99bf-a29c21453751-kube-api-access-zkzfb\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.939844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.939784    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70e57cd4-a6c4-4299-99bf-a29c21453751-installation-pull-secrets\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.940295 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.940272    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70e57cd4-a6c4-4299-99bf-a29c21453751-ca-trust-extracted\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.940646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.940629    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70e57cd4-a6c4-4299-99bf-a29c21453751-registry-certificates\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.941292 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.941268    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70e57cd4-a6c4-4299-99bf-a29c21453751-trusted-ca\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.942461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.942431    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70e57cd4-a6c4-4299-99bf-a29c21453751-installation-pull-secrets\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.942568 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.942467    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70e57cd4-a6c4-4299-99bf-a29c21453751-image-registry-private-configuration\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.942623 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.942572    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70e57cd4-a6c4-4299-99bf-a29c21453751-registry-tls\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.959002 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.958975    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70e57cd4-a6c4-4299-99bf-a29c21453751-bound-sa-token\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:35.959215 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:35.959195    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkzfb\" (UniqueName: \"kubernetes.io/projected/70e57cd4-a6c4-4299-99bf-a29c21453751-kube-api-access-zkzfb\") pod \"image-registry-dcb998c6b-xlxth\" (UID: \"70e57cd4-a6c4-4299-99bf-a29c21453751\") " pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:36.038320 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:36.038234    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:36.362952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:36.362881    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dcb998c6b-xlxth"]
Oct 29 22:14:36.366089 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:36.366059    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e57cd4_a6c4_4299_99bf_a29c21453751.slice/crio-6a2556d832d1c3ebdfb69260cf3b86e6c06e6f646ded3800bf4be5910949ae48 WatchSource:0}: Error finding container 6a2556d832d1c3ebdfb69260cf3b86e6c06e6f646ded3800bf4be5910949ae48: Status 404 returned error can't find the container with id 6a2556d832d1c3ebdfb69260cf3b86e6c06e6f646ded3800bf4be5910949ae48
Oct 29 22:14:37.336783 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:37.336742    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dcb998c6b-xlxth" event={"ID":"70e57cd4-a6c4-4299-99bf-a29c21453751","Type":"ContainerStarted","Data":"93bbbc6f5bff653ed572d2b62454e8069e5339b6f4ec98a0f177f1512cf1ff71"}
Oct 29 22:14:37.337174 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:37.336828    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dcb998c6b-xlxth" event={"ID":"70e57cd4-a6c4-4299-99bf-a29c21453751","Type":"ContainerStarted","Data":"6a2556d832d1c3ebdfb69260cf3b86e6c06e6f646ded3800bf4be5910949ae48"}
Oct 29 22:14:37.337174 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:37.336862    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:37.403465 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:37.403409    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-dcb998c6b-xlxth" podStartSLOduration=2.403396708 podStartE2EDuration="2.403396708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:37.390520118 +0000 UTC m=+16.301963682" watchObservedRunningTime="2026-10-29 22:14:37.403396708 +0000 UTC m=+16.314840262"
Oct 29 22:14:38.310588 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:38.310559    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xp49w"
Oct 29 22:14:43.162785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:43.162746    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v552v"
Oct 29 22:14:43.691346 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:14:43.691310    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:14:44.674184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.674146    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k45qw"]
Oct 29 22:14:44.680218 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.680192    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.700943 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.700916    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 29 22:14:44.713508 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.713284    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mlg6x\""
Oct 29 22:14:44.713508 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.713314    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 29 22:14:44.713722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.713529    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 22:14:44.713722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.713529    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 22:14:44.713722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.713565    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 29 22:14:44.713722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.713716    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 29 22:14:44.763437 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.763405    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlnh6"
Oct 29 22:14:44.797181 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.797140    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plg8l\" (UniqueName: \"kubernetes.io/projected/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-kube-api-access-plg8l\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.797396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.797186    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-node-exporter-accelerators-collector-config\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.797396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.797302    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-sys\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.797396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.797333    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-root\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.797396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.797350    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-node-exporter-textfile\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.797396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.797366    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-metrics-client-ca\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.797576 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.797423    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-node-exporter-wtmp\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.797576 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.797519    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-node-exporter-tls\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.797576 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.797554    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.898781 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.898743    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-node-exporter-accelerators-collector-config\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.898781 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.898799    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-sys\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.899058 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.898867    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-root\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.899058 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.898887    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-node-exporter-textfile\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.899058 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.898908    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-metrics-client-ca\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.899058 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.898941    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-node-exporter-wtmp\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.899058 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.898990    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-sys\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.899058 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.898990    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-root\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.899058 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.899000    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-node-exporter-tls\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.899389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.899073    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.899389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.899147    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plg8l\" (UniqueName: \"kubernetes.io/projected/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-kube-api-access-plg8l\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.899389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.899167    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-node-exporter-wtmp\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.899547 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.899457    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-node-exporter-accelerators-collector-config\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.899758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.899732    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-metrics-client-ca\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.900082 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.899912    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-node-exporter-textfile\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.901617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.901593    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.901710 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.901641    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-node-exporter-tls\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:44.918943 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:44.918917    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plg8l\" (UniqueName: \"kubernetes.io/projected/cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a-kube-api-access-plg8l\") pod \"node-exporter-k45qw\" (UID: \"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a\") " pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:45.000303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:45.000202    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k45qw"
Oct 29 22:14:45.030590 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:45.030549    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3d85d6_f7e5_47fd_9667_cbc0b566bf3a.slice/crio-9efd28bd5d5926a9a7e4a9f65ce614acdd22686798b1659876bf8dc6539a2424 WatchSource:0}: Error finding container 9efd28bd5d5926a9a7e4a9f65ce614acdd22686798b1659876bf8dc6539a2424: Status 404 returned error can't find the container with id 9efd28bd5d5926a9a7e4a9f65ce614acdd22686798b1659876bf8dc6539a2424
Oct 29 22:14:45.364828 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:45.364731    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k45qw" event={"ID":"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a","Type":"ContainerStarted","Data":"9efd28bd5d5926a9a7e4a9f65ce614acdd22686798b1659876bf8dc6539a2424"}
Oct 29 22:14:46.370031 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:46.369994    4559 generic.go:356] "Generic (PLEG): container finished" podID="cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a" containerID="d80b8f4b22eda0f4ef37037f163db80778b917e6c4a44a6bfd14cd3fc55776e0" exitCode=0
Oct 29 22:14:46.370031 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:46.370033    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k45qw" event={"ID":"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a","Type":"ContainerDied","Data":"d80b8f4b22eda0f4ef37037f163db80778b917e6c4a44a6bfd14cd3fc55776e0"}
Oct 29 22:14:46.829677 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:46.829642    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"]
Oct 29 22:14:46.833190 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:46.833174    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:46.850385 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:46.850355    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 29 22:14:46.850613 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:46.850595    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 29 22:14:46.850699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:46.850600    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 29 22:14:46.850699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:46.850639    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4g78z\""
Oct 29 22:14:46.862857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:46.862827    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 29 22:14:46.863048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:46.863031    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-69hbekqp4bocq\""
Oct 29 22:14:46.863098 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:46.863070    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 29 22:14:46.864053 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:46.864035    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"]
Oct 29 22:14:47.016060 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.016021    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-thanos-querier-tls\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.016060 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.016067    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03c1223b-be7a-4231-96da-695de42bad71-tmp\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.016306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.016085    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22n6s\" (UniqueName: \"kubernetes.io/projected/03c1223b-be7a-4231-96da-695de42bad71-kube-api-access-22n6s\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.016306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.016204    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.016306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.016227    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.016306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.016246    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03c1223b-be7a-4231-96da-695de42bad71-metrics-client-ca\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.016306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.016295    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-grpc-tls\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.016458 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.016338    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.016458 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.016364    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.117424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.117338    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.117424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.117389    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-thanos-querier-tls\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.117424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.117415    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03c1223b-be7a-4231-96da-695de42bad71-tmp\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.117697 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.117433    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22n6s\" (UniqueName: \"kubernetes.io/projected/03c1223b-be7a-4231-96da-695de42bad71-kube-api-access-22n6s\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.117697 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.117456    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.117697 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.117473    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.117697 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.117492    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03c1223b-be7a-4231-96da-695de42bad71-metrics-client-ca\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.117697 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.117507    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-grpc-tls\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.117697 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.117535    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.118009 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.117860    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03c1223b-be7a-4231-96da-695de42bad71-tmp\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.118717 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.118684    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03c1223b-be7a-4231-96da-695de42bad71-metrics-client-ca\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.120236 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.120183    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-thanos-querier-tls\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.120552 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.120530    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.120647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.120571    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-grpc-tls\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.120647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.120606    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.120749 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.120728    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.120790 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.120749    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03c1223b-be7a-4231-96da-695de42bad71-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.136976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.136951    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22n6s\" (UniqueName: \"kubernetes.io/projected/03c1223b-be7a-4231-96da-695de42bad71-kube-api-access-22n6s\") pod \"thanos-querier-68c6cb5b5b-hnxcz\" (UID: \"03c1223b-be7a-4231-96da-695de42bad71\") " pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.155797 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.155772    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:47.375034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.374949    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k45qw" event={"ID":"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a","Type":"ContainerStarted","Data":"0aed776ce3bdf1f5a78c43f95fb5e6b296e28076c433d093b7327092b0dc637f"}
Oct 29 22:14:47.375034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.375006    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k45qw" event={"ID":"cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a","Type":"ContainerStarted","Data":"0f01b3b101ad3907e614cbe42608ed24e34332f217a62fb4922e3515d8f71e70"}
Oct 29 22:14:47.417572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.417517    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k45qw" podStartSLOduration=2.491189319 podStartE2EDuration="3.417501662s" totalImagesPullingTime="849.828975ms" totalInitContainerRuntime="76.483368ms" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:47.417059544 +0000 UTC m=+26.328503116" watchObservedRunningTime="2026-10-29 22:14:47.417501662 +0000 UTC m=+26.328945216"
Oct 29 22:14:47.476552 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:47.476509    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"]
Oct 29 22:14:47.479147 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:47.479118    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c1223b_be7a_4231_96da_695de42bad71.slice/crio-f84c2fbe519902c257abef5ff6f74be689c9a50c9c7a32c8ab5fbb7a7f852b09 WatchSource:0}: Error finding container f84c2fbe519902c257abef5ff6f74be689c9a50c9c7a32c8ab5fbb7a7f852b09: Status 404 returned error can't find the container with id f84c2fbe519902c257abef5ff6f74be689c9a50c9c7a32c8ab5fbb7a7f852b09
Oct 29 22:14:48.380564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:48.380508    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz" event={"ID":"03c1223b-be7a-4231-96da-695de42bad71","Type":"ContainerStarted","Data":"f84c2fbe519902c257abef5ff6f74be689c9a50c9c7a32c8ab5fbb7a7f852b09"}
Oct 29 22:14:49.131529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.131492    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-58fdf66b86-ht9bn"]
Oct 29 22:14:49.134869 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.134843    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.150623 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.150591    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 29 22:14:49.163421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.163392    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mgt2w\""
Oct 29 22:14:49.163421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.163392    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 29 22:14:49.163651 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.163426    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5jd902ms0g632\""
Oct 29 22:14:49.163651 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.163515    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 29 22:14:49.163651 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.163613    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 29 22:14:49.164389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.164360    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-58fdf66b86-ht9bn"]
Oct 29 22:14:49.234406 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.234368    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlj2k\" (UniqueName: \"kubernetes.io/projected/21ba59b0-85ea-4547-be79-29c9a4008a3a-kube-api-access-wlj2k\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.234603 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.234442    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/21ba59b0-85ea-4547-be79-29c9a4008a3a-secret-metrics-server-client-certs\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.234603 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.234533    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21ba59b0-85ea-4547-be79-29c9a4008a3a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.234603 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.234565    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/21ba59b0-85ea-4547-be79-29c9a4008a3a-audit-log\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.234603 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.234583    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/21ba59b0-85ea-4547-be79-29c9a4008a3a-secret-metrics-server-tls\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.234851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.234609    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/21ba59b0-85ea-4547-be79-29c9a4008a3a-client-ca-bundle\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.234851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.234736    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/21ba59b0-85ea-4547-be79-29c9a4008a3a-metrics-server-audit-profiles\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.335887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.335802    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/21ba59b0-85ea-4547-be79-29c9a4008a3a-secret-metrics-server-client-certs\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.336047 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.335907    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21ba59b0-85ea-4547-be79-29c9a4008a3a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.336047 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.335933    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/21ba59b0-85ea-4547-be79-29c9a4008a3a-audit-log\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.336047 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.335950    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/21ba59b0-85ea-4547-be79-29c9a4008a3a-secret-metrics-server-tls\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.336047 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.335968    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/21ba59b0-85ea-4547-be79-29c9a4008a3a-client-ca-bundle\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.336047 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.335987    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/21ba59b0-85ea-4547-be79-29c9a4008a3a-metrics-server-audit-profiles\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.336289 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.336070    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlj2k\" (UniqueName: \"kubernetes.io/projected/21ba59b0-85ea-4547-be79-29c9a4008a3a-kube-api-access-wlj2k\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.336412 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.336351    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/21ba59b0-85ea-4547-be79-29c9a4008a3a-audit-log\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.336777 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.336749    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21ba59b0-85ea-4547-be79-29c9a4008a3a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.337004 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.336986    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/21ba59b0-85ea-4547-be79-29c9a4008a3a-metrics-server-audit-profiles\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.338512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.338484    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/21ba59b0-85ea-4547-be79-29c9a4008a3a-secret-metrics-server-client-certs\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.338667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.338648    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/21ba59b0-85ea-4547-be79-29c9a4008a3a-secret-metrics-server-tls\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.338742 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.338725    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/21ba59b0-85ea-4547-be79-29c9a4008a3a-client-ca-bundle\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.356240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.356212    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlj2k\" (UniqueName: \"kubernetes.io/projected/21ba59b0-85ea-4547-be79-29c9a4008a3a-kube-api-access-wlj2k\") pod \"metrics-server-58fdf66b86-ht9bn\" (UID: \"21ba59b0-85ea-4547-be79-29c9a4008a3a\") " pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.402676 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.402648    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-87d9bdc54-dt5tk"]
Oct 29 22:14:49.406928 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.406911    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.429783 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.429735    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 29 22:14:49.429933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.429738    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 29 22:14:49.430058 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.430036    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 29 22:14:49.430058 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.430035    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 29 22:14:49.430240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.430086    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 29 22:14:49.430240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.430153    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 29 22:14:49.430240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.430205    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6mtnl\""
Oct 29 22:14:49.430380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.430273    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87d9bdc54-dt5tk"]
Oct 29 22:14:49.448869 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.448843    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:14:49.455836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.455181    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 29 22:14:49.455836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.455254    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 29 22:14:49.537278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.537254    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t96l\" (UniqueName: \"kubernetes.io/projected/788439ed-27a0-4816-b4de-1bb940e02d50-kube-api-access-9t96l\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.537408 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.537289    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/788439ed-27a0-4816-b4de-1bb940e02d50-tmp\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.537408 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.537385    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/788439ed-27a0-4816-b4de-1bb940e02d50-console-serving-cert\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.537490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.537414    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-console-config\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.537490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.537449    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-trusted-ca-bundle\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.537567 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.537509    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/788439ed-27a0-4816-b4de-1bb940e02d50-console-oauth-config\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.537620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.537595    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-oauth-serving-cert\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.537657 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.537640    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-service-ca\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.638126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.638040    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/788439ed-27a0-4816-b4de-1bb940e02d50-console-serving-cert\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.638126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.638077    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-console-config\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.638353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.638192    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-trusted-ca-bundle\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.638353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.638229    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/788439ed-27a0-4816-b4de-1bb940e02d50-console-oauth-config\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.638353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.638262    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-oauth-serving-cert\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.638353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.638286    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-service-ca\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.638353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.638332    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t96l\" (UniqueName: \"kubernetes.io/projected/788439ed-27a0-4816-b4de-1bb940e02d50-kube-api-access-9t96l\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.638587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.638362    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/788439ed-27a0-4816-b4de-1bb940e02d50-tmp\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.638751 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.638724    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/788439ed-27a0-4816-b4de-1bb940e02d50-tmp\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.638929 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.638855    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-console-config\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.639175 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.639139    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-service-ca\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.639287 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.639246    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-trusted-ca-bundle\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.639333 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.639295    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-oauth-serving-cert\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.640686 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.640667    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/788439ed-27a0-4816-b4de-1bb940e02d50-console-serving-cert\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.640766 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.640744    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/788439ed-27a0-4816-b4de-1bb940e02d50-console-oauth-config\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.658067 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.658033    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t96l\" (UniqueName: \"kubernetes.io/projected/788439ed-27a0-4816-b4de-1bb940e02d50-kube-api-access-9t96l\") pod \"console-87d9bdc54-dt5tk\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") " pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.725497 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.725458    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:49.769950 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:49.769923    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-58fdf66b86-ht9bn"]
Oct 29 22:14:49.775287 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:49.773754    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ba59b0_85ea_4547_be79_29c9a4008a3a.slice/crio-4b3d80c85a9333a2f0059e4cdcbefa6e4c195f6224a3aef0b82321a168d03cfc WatchSource:0}: Error finding container 4b3d80c85a9333a2f0059e4cdcbefa6e4c195f6224a3aef0b82321a168d03cfc: Status 404 returned error can't find the container with id 4b3d80c85a9333a2f0059e4cdcbefa6e4c195f6224a3aef0b82321a168d03cfc
Oct 29 22:14:50.022756 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:50.022720    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87d9bdc54-dt5tk"]
Oct 29 22:14:50.026224 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:14:50.026192    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788439ed_27a0_4816_b4de_1bb940e02d50.slice/crio-33159ba9a179b3778f2dedb12e4ac92e485c750fb16cde5a8e62f1bf7897d3d8 WatchSource:0}: Error finding container 33159ba9a179b3778f2dedb12e4ac92e485c750fb16cde5a8e62f1bf7897d3d8: Status 404 returned error can't find the container with id 33159ba9a179b3778f2dedb12e4ac92e485c750fb16cde5a8e62f1bf7897d3d8
Oct 29 22:14:50.389967 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:50.389926    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn" event={"ID":"21ba59b0-85ea-4547-be79-29c9a4008a3a","Type":"ContainerStarted","Data":"4b3d80c85a9333a2f0059e4cdcbefa6e4c195f6224a3aef0b82321a168d03cfc"}
Oct 29 22:14:50.392795 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:50.392752    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz" event={"ID":"03c1223b-be7a-4231-96da-695de42bad71","Type":"ContainerStarted","Data":"85ecc5cd9e8d4fc242ce0fe714afb15dae4afc234e62d13f650311fb35b87181"}
Oct 29 22:14:50.392943 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:50.392885    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz" event={"ID":"03c1223b-be7a-4231-96da-695de42bad71","Type":"ContainerStarted","Data":"a4c4298bc5b6698d773e53f84430b6501fb4086598fecd581c6a889a6b23bbf9"}
Oct 29 22:14:50.392943 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:50.392934    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz" event={"ID":"03c1223b-be7a-4231-96da-695de42bad71","Type":"ContainerStarted","Data":"a06be8c63473f6c8c7e7f3ac0fce7821e6bb6d4c66fbab6c9f0e0e3478729663"}
Oct 29 22:14:50.393033 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:50.392974    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz" event={"ID":"03c1223b-be7a-4231-96da-695de42bad71","Type":"ContainerStarted","Data":"b5d5b689d2c8856d4a466cbd17587d4bb90378c6488e6bddacd2ed390e2d6059"}
Oct 29 22:14:50.394428 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:50.394405    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87d9bdc54-dt5tk" event={"ID":"788439ed-27a0-4816-b4de-1bb940e02d50","Type":"ContainerStarted","Data":"33159ba9a179b3778f2dedb12e4ac92e485c750fb16cde5a8e62f1bf7897d3d8"}
Oct 29 22:14:51.401987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:51.401944    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz" event={"ID":"03c1223b-be7a-4231-96da-695de42bad71","Type":"ContainerStarted","Data":"bdd6c0deab734d08c77d5071c4f982e46daf30a35d60f6e9852efa0828bbf6c8"}
Oct 29 22:14:51.402440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:51.402020    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz" event={"ID":"03c1223b-be7a-4231-96da-695de42bad71","Type":"ContainerStarted","Data":"f60ef8a77caa845003f3a949c554124d746598f9780e36c62d46f164a981ad25"}
Oct 29 22:14:51.402440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:51.402125    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:51.473678 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:51.473626    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz" podStartSLOduration=2.89865154 podStartE2EDuration="5.473613201s" totalImagesPullingTime="2.574961661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:51.460781861 +0000 UTC m=+30.372225416" watchObservedRunningTime="2026-10-29 22:14:51.473613201 +0000 UTC m=+30.385056755"
Oct 29 22:14:52.409617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:52.409577    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn" event={"ID":"21ba59b0-85ea-4547-be79-29c9a4008a3a","Type":"ContainerStarted","Data":"ccd8780f3d669fc40e3662b05ff7cd256e8cd4e9ba9b95034fb21afc886cb980"}
Oct 29 22:14:52.452038 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:52.451975    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn" podStartSLOduration=1.646302002 podStartE2EDuration="3.451959702s" totalImagesPullingTime="1.8056577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:52.450591792 +0000 UTC m=+31.362035346" watchObservedRunningTime="2026-10-29 22:14:52.451959702 +0000 UTC m=+31.363403252"
Oct 29 22:14:53.414832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:53.414774    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87d9bdc54-dt5tk" event={"ID":"788439ed-27a0-4816-b4de-1bb940e02d50","Type":"ContainerStarted","Data":"5138d4595cf8787c90af87a10aa4fd680f628f8f6773d43aff80fa001c3863c5"}
Oct 29 22:14:54.459082 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:54.459026    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-87d9bdc54-dt5tk" podStartSLOduration=2.15927824 podStartE2EDuration="5.45901176s" totalImagesPullingTime="3.29973352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:54.457985611 +0000 UTC m=+33.369429165" watchObservedRunningTime="2026-10-29 22:14:54.45901176 +0000 UTC m=+33.370455385"
Oct 29 22:14:57.415963 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:57.415933    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-68c6cb5b5b-hnxcz"
Oct 29 22:14:58.344503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:58.344472    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-dcb998c6b-xlxth"
Oct 29 22:14:58.520644 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:58.520610    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5pq4b"
Oct 29 22:14:59.726462 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:59.726427    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:59.726886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:59.726487    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:14:59.731467 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:14:59.731441    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:15:00.423824 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:00.423776    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-87d9bdc54-dt5tk"]
Oct 29 22:15:00.445595 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:00.445563    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:15:09.449493 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:09.449408    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:15:09.449493 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:09.449471    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:15:13.691934 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:15:13.691894    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:15:26.464001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.463940    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-87d9bdc54-dt5tk" podUID="788439ed-27a0-4816-b4de-1bb940e02d50" containerName="console" containerID="cri-o://5138d4595cf8787c90af87a10aa4fd680f628f8f6773d43aff80fa001c3863c5" gracePeriod=15
Oct 29 22:15:26.704877 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.704852    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:15:26.809118 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.809027    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-console-config\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-console-config\") pod \"788439ed-27a0-4816-b4de-1bb940e02d50\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") "
Oct 29 22:15:26.809118 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.809073    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/788439ed-27a0-4816-b4de-1bb940e02d50-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/788439ed-27a0-4816-b4de-1bb940e02d50-console-oauth-config\") pod \"788439ed-27a0-4816-b4de-1bb940e02d50\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") "
Oct 29 22:15:26.809118 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.809100    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-service-ca\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-service-ca\") pod \"788439ed-27a0-4816-b4de-1bb940e02d50\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") "
Oct 29 22:15:26.809337 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.809253    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788439ed-27a0-4816-b4de-1bb940e02d50-tmp\" (UniqueName: \"kubernetes.io/empty-dir/788439ed-27a0-4816-b4de-1bb940e02d50-tmp\") pod \"788439ed-27a0-4816-b4de-1bb940e02d50\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") "
Oct 29 22:15:26.809337 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.809303    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788439ed-27a0-4816-b4de-1bb940e02d50-kube-api-access-9t96l\" (UniqueName: \"kubernetes.io/projected/788439ed-27a0-4816-b4de-1bb940e02d50-kube-api-access-9t96l\") pod \"788439ed-27a0-4816-b4de-1bb940e02d50\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") "
Oct 29 22:15:26.809337 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.809330    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-trusted-ca-bundle\") pod \"788439ed-27a0-4816-b4de-1bb940e02d50\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") "
Oct 29 22:15:26.809495 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.809371    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/788439ed-27a0-4816-b4de-1bb940e02d50-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/788439ed-27a0-4816-b4de-1bb940e02d50-console-serving-cert\") pod \"788439ed-27a0-4816-b4de-1bb940e02d50\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") "
Oct 29 22:15:26.809495 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.809423    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-oauth-serving-cert\") pod \"788439ed-27a0-4816-b4de-1bb940e02d50\" (UID: \"788439ed-27a0-4816-b4de-1bb940e02d50\") "
Oct 29 22:15:26.809600 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.809544    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-console-config" pod "788439ed-27a0-4816-b4de-1bb940e02d50" (UID: "788439ed-27a0-4816-b4de-1bb940e02d50"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:26.809600 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.809559    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-service-ca" pod "788439ed-27a0-4816-b4de-1bb940e02d50" (UID: "788439ed-27a0-4816-b4de-1bb940e02d50"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:26.809700 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.809676    4559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:15:26.809700 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.809694    4559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-console-config\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:15:26.809837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.809771    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788439ed-27a0-4816-b4de-1bb940e02d50-tmp" pod "788439ed-27a0-4816-b4de-1bb940e02d50" (UID: "788439ed-27a0-4816-b4de-1bb940e02d50"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:26.809837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.809680    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-trusted-ca-bundle" pod "788439ed-27a0-4816-b4de-1bb940e02d50" (UID: "788439ed-27a0-4816-b4de-1bb940e02d50"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:26.809952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.809848    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-oauth-serving-cert" pod "788439ed-27a0-4816-b4de-1bb940e02d50" (UID: "788439ed-27a0-4816-b4de-1bb940e02d50"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:26.811578 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.811553    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/788439ed-27a0-4816-b4de-1bb940e02d50-console-oauth-config" pod "788439ed-27a0-4816-b4de-1bb940e02d50" (UID: "788439ed-27a0-4816-b4de-1bb940e02d50"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:26.811709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.811591    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788439ed-27a0-4816-b4de-1bb940e02d50-kube-api-access-9t96l" pod "788439ed-27a0-4816-b4de-1bb940e02d50" (UID: "788439ed-27a0-4816-b4de-1bb940e02d50"). InnerVolumeSpecName "kube-api-access-9t96l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:26.811709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.811659    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/788439ed-27a0-4816-b4de-1bb940e02d50-console-serving-cert" pod "788439ed-27a0-4816-b4de-1bb940e02d50" (UID: "788439ed-27a0-4816-b4de-1bb940e02d50"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:26.910450 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.910397    4559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/788439ed-27a0-4816-b4de-1bb940e02d50-console-oauth-config\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:15:26.910450 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.910442    4559 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/788439ed-27a0-4816-b4de-1bb940e02d50-tmp\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:15:26.910669 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.910454    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t96l\" (UniqueName: \"kubernetes.io/projected/788439ed-27a0-4816-b4de-1bb940e02d50-kube-api-access-9t96l\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:15:26.910669 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.910491    4559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-trusted-ca-bundle\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:15:26.910669 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.910500    4559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/788439ed-27a0-4816-b4de-1bb940e02d50-console-serving-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:15:26.910669 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:26.910508    4559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/788439ed-27a0-4816-b4de-1bb940e02d50-oauth-serving-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:15:27.537230 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:27.537141    4559 generic.go:356] "Generic (PLEG): container finished" podID="788439ed-27a0-4816-b4de-1bb940e02d50" containerID="5138d4595cf8787c90af87a10aa4fd680f628f8f6773d43aff80fa001c3863c5" exitCode=0
Oct 29 22:15:27.537230 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:27.537195    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87d9bdc54-dt5tk"
Oct 29 22:15:27.537230 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:27.537213    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87d9bdc54-dt5tk" event={"ID":"788439ed-27a0-4816-b4de-1bb940e02d50","Type":"ContainerDied","Data":"5138d4595cf8787c90af87a10aa4fd680f628f8f6773d43aff80fa001c3863c5"}
Oct 29 22:15:27.537736 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:27.537263    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87d9bdc54-dt5tk" event={"ID":"788439ed-27a0-4816-b4de-1bb940e02d50","Type":"ContainerDied","Data":"33159ba9a179b3778f2dedb12e4ac92e485c750fb16cde5a8e62f1bf7897d3d8"}
Oct 29 22:15:27.537736 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:27.537290    4559 scope.go:122] "RemoveContainer" containerID="5138d4595cf8787c90af87a10aa4fd680f628f8f6773d43aff80fa001c3863c5"
Oct 29 22:15:27.545424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:27.545401    4559 scope.go:122] "RemoveContainer" containerID="5138d4595cf8787c90af87a10aa4fd680f628f8f6773d43aff80fa001c3863c5"
Oct 29 22:15:27.545710 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:15:27.545683    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5138d4595cf8787c90af87a10aa4fd680f628f8f6773d43aff80fa001c3863c5\": container with ID starting with 5138d4595cf8787c90af87a10aa4fd680f628f8f6773d43aff80fa001c3863c5 not found: ID does not exist" containerID="5138d4595cf8787c90af87a10aa4fd680f628f8f6773d43aff80fa001c3863c5"
Oct 29 22:15:27.545827 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:27.545718    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5138d4595cf8787c90af87a10aa4fd680f628f8f6773d43aff80fa001c3863c5"} err="failed to get container status \"5138d4595cf8787c90af87a10aa4fd680f628f8f6773d43aff80fa001c3863c5\": rpc error: code = NotFound desc = could not find container \"5138d4595cf8787c90af87a10aa4fd680f628f8f6773d43aff80fa001c3863c5\": container with ID starting with 5138d4595cf8787c90af87a10aa4fd680f628f8f6773d43aff80fa001c3863c5 not found: ID does not exist"
Oct 29 22:15:27.588229 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:27.588193    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-87d9bdc54-dt5tk"]
Oct 29 22:15:27.601001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:27.600937    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-87d9bdc54-dt5tk"]
Oct 29 22:15:29.228477 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:29.228441    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="788439ed-27a0-4816-b4de-1bb940e02d50" path="/var/lib/kubelet/pods/788439ed-27a0-4816-b4de-1bb940e02d50/volumes"
Oct 29 22:15:29.455902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:29.455866    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:15:29.459797 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:29.459767    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn"
Oct 29 22:15:30.875276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.875241    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-78458cfc6-sx8jf"]
Oct 29 22:15:30.875759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.875609    4559 cpu_manager.go:414] "containerMap: removing container" podUID="788439ed-27a0-4816-b4de-1bb940e02d50" containerName="console"
Oct 29 22:15:30.875759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.875627    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="788439ed-27a0-4816-b4de-1bb940e02d50" containerName="console"
Oct 29 22:15:30.875759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.875732    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="788439ed-27a0-4816-b4de-1bb940e02d50" containerName="console"
Oct 29 22:15:30.878596 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.878574    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:30.894498 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.894473    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 29 22:15:30.897525 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.897508    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 29 22:15:30.897801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.897784    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 29 22:15:30.907580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.907549    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6mtnl\""
Oct 29 22:15:30.907726 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.907631    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 29 22:15:30.911356 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.911332    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78458cfc6-sx8jf"]
Oct 29 22:15:30.939699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.939658    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b5f3cca-319e-4bd3-84de-f054f82473c2-console-oauth-config\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:30.939699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.939699    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b5f3cca-319e-4bd3-84de-f054f82473c2-tmp\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:30.939993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.939722    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b5f3cca-319e-4bd3-84de-f054f82473c2-oauth-serving-cert\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:30.939993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.939786    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b5f3cca-319e-4bd3-84de-f054f82473c2-console-serving-cert\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:30.940108 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.939994    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b5f3cca-319e-4bd3-84de-f054f82473c2-console-config\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:30.940108 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.940045    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b5f3cca-319e-4bd3-84de-f054f82473c2-trusted-ca-bundle\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:30.940214 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.940114    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5cgm\" (UniqueName: \"kubernetes.io/projected/7b5f3cca-319e-4bd3-84de-f054f82473c2-kube-api-access-p5cgm\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:30.940214 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.940189    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5f3cca-319e-4bd3-84de-f054f82473c2-service-ca\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:30.995102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.995065    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 29 22:15:30.995102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.995080    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 29 22:15:30.995102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:30.995089    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 29 22:15:31.021298 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.021269    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 29 22:15:31.041525 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.041491    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b5f3cca-319e-4bd3-84de-f054f82473c2-trusted-ca-bundle\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:31.041615 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.041530    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5cgm\" (UniqueName: \"kubernetes.io/projected/7b5f3cca-319e-4bd3-84de-f054f82473c2-kube-api-access-p5cgm\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:31.041615 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.041561    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5f3cca-319e-4bd3-84de-f054f82473c2-service-ca\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:31.041721 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.041700    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b5f3cca-319e-4bd3-84de-f054f82473c2-console-oauth-config\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:31.041775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.041740    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b5f3cca-319e-4bd3-84de-f054f82473c2-tmp\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:31.041858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.041776    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b5f3cca-319e-4bd3-84de-f054f82473c2-oauth-serving-cert\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:31.041858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.041805    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b5f3cca-319e-4bd3-84de-f054f82473c2-console-serving-cert\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:31.041962 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.041911    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b5f3cca-319e-4bd3-84de-f054f82473c2-console-config\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:31.042236 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.042201    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b5f3cca-319e-4bd3-84de-f054f82473c2-tmp\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:31.042405 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.042386    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5f3cca-319e-4bd3-84de-f054f82473c2-service-ca\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:31.042537 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.042514    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b5f3cca-319e-4bd3-84de-f054f82473c2-trusted-ca-bundle\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:31.042605 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.042555    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b5f3cca-319e-4bd3-84de-f054f82473c2-console-config\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:31.042835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.042789    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b5f3cca-319e-4bd3-84de-f054f82473c2-oauth-serving-cert\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:31.044424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.044403    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b5f3cca-319e-4bd3-84de-f054f82473c2-console-serving-cert\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:31.044505 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.044435    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b5f3cca-319e-4bd3-84de-f054f82473c2-console-oauth-config\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:31.061917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.061889    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5cgm\" (UniqueName: \"kubernetes.io/projected/7b5f3cca-319e-4bd3-84de-f054f82473c2-kube-api-access-p5cgm\") pod \"console-78458cfc6-sx8jf\" (UID: \"7b5f3cca-319e-4bd3-84de-f054f82473c2\") " pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:31.190342 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.190301    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:31.489753 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.489730    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78458cfc6-sx8jf"]
Oct 29 22:15:31.492597 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:15:31.492570    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5f3cca_319e_4bd3_84de_f054f82473c2.slice/crio-ec75885024a61102f0b942d5ed2df4488c01589e08df6e62093959b65c3a8897 WatchSource:0}: Error finding container ec75885024a61102f0b942d5ed2df4488c01589e08df6e62093959b65c3a8897: Status 404 returned error can't find the container with id ec75885024a61102f0b942d5ed2df4488c01589e08df6e62093959b65c3a8897
Oct 29 22:15:31.553630 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:31.553598    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78458cfc6-sx8jf" event={"ID":"7b5f3cca-319e-4bd3-84de-f054f82473c2","Type":"ContainerStarted","Data":"ec75885024a61102f0b942d5ed2df4488c01589e08df6e62093959b65c3a8897"}
Oct 29 22:15:32.559005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:32.558968    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78458cfc6-sx8jf" event={"ID":"7b5f3cca-319e-4bd3-84de-f054f82473c2","Type":"ContainerStarted","Data":"600ae527475619b42a8a7dd1b0b017afce54c215484b4ded40e088adcbec53ba"}
Oct 29 22:15:32.600710 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:32.600651    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-78458cfc6-sx8jf" podStartSLOduration=2.600636422 podStartE2EDuration="2.600636422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:32.59989589 +0000 UTC m=+71.511339444" watchObservedRunningTime="2026-10-29 22:15:32.600636422 +0000 UTC m=+71.512079976"
Oct 29 22:15:37.687193 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.687155    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 22:15:37.691172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.691154    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.707092 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.707058    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 29 22:15:37.707092 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.707075    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 29 22:15:37.707313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.707095    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 29 22:15:37.707313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.707102    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 29 22:15:37.707313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.707077    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 29 22:15:37.707313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.707057    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 29 22:15:37.707313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.707142    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 29 22:15:37.707313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.707281    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5fq7m\""
Oct 29 22:15:37.739167 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.739139    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 29 22:15:37.740129 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.740106    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 22:15:37.793946 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.793914    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.793946 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.793949    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-web-config\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.794232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.793969    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/851cfdee-e076-421a-a269-c03c3fdaf5ce-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.794232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.793990    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/851cfdee-e076-421a-a269-c03c3fdaf5ce-config-out\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.794232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.794081    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/851cfdee-e076-421a-a269-c03c3fdaf5ce-tls-assets\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.794232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.794129    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.794429 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.794245    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.794429 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.794326    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-config-volume\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.794429 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.794356    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/851cfdee-e076-421a-a269-c03c3fdaf5ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.794429 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.794388    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbrln\" (UniqueName: \"kubernetes.io/projected/851cfdee-e076-421a-a269-c03c3fdaf5ce-kube-api-access-zbrln\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.794551 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.794454    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/851cfdee-e076-421a-a269-c03c3fdaf5ce-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.794551 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.794490    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.794551 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.794515    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.807363 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.807341    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 29 22:15:37.895926 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.895893    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.896048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.895945    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-config-volume\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.896048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.895961    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/851cfdee-e076-421a-a269-c03c3fdaf5ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.896048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.895980    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbrln\" (UniqueName: \"kubernetes.io/projected/851cfdee-e076-421a-a269-c03c3fdaf5ce-kube-api-access-zbrln\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.896048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.895995    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/851cfdee-e076-421a-a269-c03c3fdaf5ce-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.896260 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.896038    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.896260 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.896092    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.896260 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.896138    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.896260 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.896162    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-web-config\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.896260 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.896187    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/851cfdee-e076-421a-a269-c03c3fdaf5ce-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.896260 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.896217    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/851cfdee-e076-421a-a269-c03c3fdaf5ce-config-out\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.896260 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.896246    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/851cfdee-e076-421a-a269-c03c3fdaf5ce-tls-assets\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.896610 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.896270    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.896610 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.896578    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/851cfdee-e076-421a-a269-c03c3fdaf5ce-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.897420 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.897330    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/851cfdee-e076-421a-a269-c03c3fdaf5ce-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.898132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.898108    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/851cfdee-e076-421a-a269-c03c3fdaf5ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.899411 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.899388    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.899857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.899791    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.899857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.899850    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-config-volume\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.900027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.899864    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.900257 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.900234    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.900326 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.900283    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.900474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.900453    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/851cfdee-e076-421a-a269-c03c3fdaf5ce-config-out\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.900956 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.900938    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/851cfdee-e076-421a-a269-c03c3fdaf5ce-tls-assets\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.901325 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.901306    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/851cfdee-e076-421a-a269-c03c3fdaf5ce-web-config\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:37.915955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:37.915925    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbrln\" (UniqueName: \"kubernetes.io/projected/851cfdee-e076-421a-a269-c03c3fdaf5ce-kube-api-access-zbrln\") pod \"alertmanager-main-0\" (UID: \"851cfdee-e076-421a-a269-c03c3fdaf5ce\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:38.015727 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:38.015625    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 29 22:15:38.347767 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:38.347658    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 22:15:38.350434 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:15:38.350408    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851cfdee_e076_421a_a269_c03c3fdaf5ce.slice/crio-573a5114994d37274c3c72d8dd54b155ee4ea6694d0647cf9020a12c071b90f1 WatchSource:0}: Error finding container 573a5114994d37274c3c72d8dd54b155ee4ea6694d0647cf9020a12c071b90f1: Status 404 returned error can't find the container with id 573a5114994d37274c3c72d8dd54b155ee4ea6694d0647cf9020a12c071b90f1
Oct 29 22:15:38.579956 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:38.579918    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"851cfdee-e076-421a-a269-c03c3fdaf5ce","Type":"ContainerStarted","Data":"573a5114994d37274c3c72d8dd54b155ee4ea6694d0647cf9020a12c071b90f1"}
Oct 29 22:15:39.585210 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:39.585173    4559 generic.go:356] "Generic (PLEG): container finished" podID="851cfdee-e076-421a-a269-c03c3fdaf5ce" containerID="fb7b81d86ac55094af269a9c7dead12542a73cfd475f3b4ee311dea9373fbcad" exitCode=0
Oct 29 22:15:39.585580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:39.585262    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"851cfdee-e076-421a-a269-c03c3fdaf5ce","Type":"ContainerDied","Data":"fb7b81d86ac55094af269a9c7dead12542a73cfd475f3b4ee311dea9373fbcad"}
Oct 29 22:15:41.191226 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:41.191192    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:41.191599 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:41.191245    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:41.196101 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:41.196080    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:41.596050 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:41.596014    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"851cfdee-e076-421a-a269-c03c3fdaf5ce","Type":"ContainerStarted","Data":"c7ee5ad5ce7166d28786e868ba514b7dd3e9fa16a5b0a86146f6ba45eb367a8c"}
Oct 29 22:15:41.596221 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:41.596080    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"851cfdee-e076-421a-a269-c03c3fdaf5ce","Type":"ContainerStarted","Data":"a9e62b880aa34b0566c61903fd64dba6184ce8f7a7bf82e3fafd0435254d7ed7"}
Oct 29 22:15:41.596221 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:41.596121    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"851cfdee-e076-421a-a269-c03c3fdaf5ce","Type":"ContainerStarted","Data":"6f86df63acb47128cec30bfdb537b21d0d8730cf7335beb80cc93a54a353947c"}
Oct 29 22:15:41.596221 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:41.596153    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"851cfdee-e076-421a-a269-c03c3fdaf5ce","Type":"ContainerStarted","Data":"4554a648f3e13098de753aff0a0dfdbc66d0d804cd74286e6b58a2721e85fdec"}
Oct 29 22:15:41.596221 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:41.596175    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"851cfdee-e076-421a-a269-c03c3fdaf5ce","Type":"ContainerStarted","Data":"b48f5b6f4b7c70a5664c92bde7fd9746257186ec79c59713a0fdda8d54f0b3e0"}
Oct 29 22:15:41.596221 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:41.596201    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"851cfdee-e076-421a-a269-c03c3fdaf5ce","Type":"ContainerStarted","Data":"3b522cd6e9378a68c15fb2349224c162499bbd0f7d6f21ab85ce50e730274ab3"}
Oct 29 22:15:41.599901 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:41.599879    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78458cfc6-sx8jf"
Oct 29 22:15:41.645522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:41.645466    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.9887113840000001 podStartE2EDuration="4.645452447s" totalImagesPullingTime="2.643002179s" totalInitContainerRuntime="13.738884ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:41.644781897 +0000 UTC m=+80.556225451" watchObservedRunningTime="2026-10-29 22:15:41.645452447 +0000 UTC m=+80.556896001"
Oct 29 22:15:42.601574 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:42.601537    4559 generic.go:356] "Generic (PLEG): container finished" podID="807ed4de-272f-4bb2-bf26-cc33f6967771" containerID="6c4cf8f1da814c401541accca948236cc4902ac0127d909c7e8595b448c3f902" exitCode=0
Oct 29 22:15:42.601978 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:42.601610    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hc4lm-rqsvk" event={"ID":"807ed4de-272f-4bb2-bf26-cc33f6967771","Type":"ContainerDied","Data":"6c4cf8f1da814c401541accca948236cc4902ac0127d909c7e8595b448c3f902"}
Oct 29 22:15:43.693871 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:15:43.693830    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:15:43.739902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:43.739880    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:15:43.852062 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:43.852027    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/807ed4de-272f-4bb2-bf26-cc33f6967771-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/807ed4de-272f-4bb2-bf26-cc33f6967771-archives-path\") pod \"807ed4de-272f-4bb2-bf26-cc33f6967771\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") "
Oct 29 22:15:43.852269 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:43.852104    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-trusted-ca-bundle\") pod \"807ed4de-272f-4bb2-bf26-cc33f6967771\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") "
Oct 29 22:15:43.852269 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:43.852136    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/807ed4de-272f-4bb2-bf26-cc33f6967771-kube-api-access-kq8zc\" (UniqueName: \"kubernetes.io/projected/807ed4de-272f-4bb2-bf26-cc33f6967771-kube-api-access-kq8zc\") pod \"807ed4de-272f-4bb2-bf26-cc33f6967771\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") "
Oct 29 22:15:43.852269 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:43.852160    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-service-ca-bundle\") pod \"807ed4de-272f-4bb2-bf26-cc33f6967771\" (UID: \"807ed4de-272f-4bb2-bf26-cc33f6967771\") "
Oct 29 22:15:43.852499 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:43.852469    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807ed4de-272f-4bb2-bf26-cc33f6967771-archives-path" pod "807ed4de-272f-4bb2-bf26-cc33f6967771" (UID: "807ed4de-272f-4bb2-bf26-cc33f6967771"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:43.852564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:43.852535    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-service-ca-bundle" pod "807ed4de-272f-4bb2-bf26-cc33f6967771" (UID: "807ed4de-272f-4bb2-bf26-cc33f6967771"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:43.852564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:43.852530    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-trusted-ca-bundle" pod "807ed4de-272f-4bb2-bf26-cc33f6967771" (UID: "807ed4de-272f-4bb2-bf26-cc33f6967771"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:43.854544 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:43.854524    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807ed4de-272f-4bb2-bf26-cc33f6967771-kube-api-access-kq8zc" pod "807ed4de-272f-4bb2-bf26-cc33f6967771" (UID: "807ed4de-272f-4bb2-bf26-cc33f6967771"). InnerVolumeSpecName "kube-api-access-kq8zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:43.953194 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:43.953157    4559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-trusted-ca-bundle\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:15:43.953194 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:43.953185    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq8zc\" (UniqueName: \"kubernetes.io/projected/807ed4de-272f-4bb2-bf26-cc33f6967771-kube-api-access-kq8zc\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:15:43.953194 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:43.953197    4559 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807ed4de-272f-4bb2-bf26-cc33f6967771-service-ca-bundle\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:15:43.953194 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:43.953205    4559 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/807ed4de-272f-4bb2-bf26-cc33f6967771-archives-path\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:15:44.609787 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:44.609752    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hc4lm-rqsvk" event={"ID":"807ed4de-272f-4bb2-bf26-cc33f6967771","Type":"ContainerDied","Data":"b7ddd0d866bb33c277ffe06ba1ea76a766bc499124e08a77ebfdf1a4d28ba0d5"}
Oct 29 22:15:44.609787 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:44.609776    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hc4lm-rqsvk"
Oct 29 22:15:44.610020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:15:44.609825    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7ddd0d866bb33c277ffe06ba1ea76a766bc499124e08a77ebfdf1a4d28ba0d5"
Oct 29 22:16:13.692089 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:16:13.692054    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:16:43.691543 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:16:43.691452    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:17:13.692531 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:17:13.692493    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:17:43.692511 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:17:43.692468    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:18:13.692384 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:18:13.692301    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:18:14.802664 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:14.802574    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-7fkbf"]
Oct 29 22:18:14.803039 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:14.802958    4559 cpu_manager.go:414] "containerMap: removing container" podUID="807ed4de-272f-4bb2-bf26-cc33f6967771" containerName="insights-gathering"
Oct 29 22:18:14.803039 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:14.802976    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="807ed4de-272f-4bb2-bf26-cc33f6967771" containerName="insights-gathering"
Oct 29 22:18:14.803113 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:14.803061    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="807ed4de-272f-4bb2-bf26-cc33f6967771" containerName="insights-gathering"
Oct 29 22:18:14.805933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:14.805918    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7fkbf"
Oct 29 22:18:14.828598 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:14.828569    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 29 22:18:14.834539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:14.834506    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-4xsdd\""
Oct 29 22:18:14.834694 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:14.834560    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 29 22:18:14.835637 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:14.835613    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-7fkbf"]
Oct 29 22:18:14.943615 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:14.943578    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2d52689-4162-48ee-abeb-84e3f606f656-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-7fkbf\" (UID: \"c2d52689-4162-48ee-abeb-84e3f606f656\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7fkbf"
Oct 29 22:18:14.943836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:14.943637    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xkv9\" (UniqueName: \"kubernetes.io/projected/c2d52689-4162-48ee-abeb-84e3f606f656-kube-api-access-9xkv9\") pod \"cert-manager-cainjector-5c57bbd4fc-7fkbf\" (UID: \"c2d52689-4162-48ee-abeb-84e3f606f656\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7fkbf"
Oct 29 22:18:15.045081 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:15.045037    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xkv9\" (UniqueName: \"kubernetes.io/projected/c2d52689-4162-48ee-abeb-84e3f606f656-kube-api-access-9xkv9\") pod \"cert-manager-cainjector-5c57bbd4fc-7fkbf\" (UID: \"c2d52689-4162-48ee-abeb-84e3f606f656\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7fkbf"
Oct 29 22:18:15.045283 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:15.045139    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2d52689-4162-48ee-abeb-84e3f606f656-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-7fkbf\" (UID: \"c2d52689-4162-48ee-abeb-84e3f606f656\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7fkbf"
Oct 29 22:18:15.070036 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:15.069978    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2d52689-4162-48ee-abeb-84e3f606f656-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-7fkbf\" (UID: \"c2d52689-4162-48ee-abeb-84e3f606f656\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7fkbf"
Oct 29 22:18:15.070255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:15.070233    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xkv9\" (UniqueName: \"kubernetes.io/projected/c2d52689-4162-48ee-abeb-84e3f606f656-kube-api-access-9xkv9\") pod \"cert-manager-cainjector-5c57bbd4fc-7fkbf\" (UID: \"c2d52689-4162-48ee-abeb-84e3f606f656\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7fkbf"
Oct 29 22:18:15.136146 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:15.136108    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7fkbf"
Oct 29 22:18:15.430907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:15.430768    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-7fkbf"]
Oct 29 22:18:15.433538 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:18:15.433513    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2d52689_4162_48ee_abeb_84e3f606f656.slice/crio-8578c23178c927d7afa6d5b24eb0a44487a317f81cab95fa40235d94e30972dc WatchSource:0}: Error finding container 8578c23178c927d7afa6d5b24eb0a44487a317f81cab95fa40235d94e30972dc: Status 404 returned error can't find the container with id 8578c23178c927d7afa6d5b24eb0a44487a317f81cab95fa40235d94e30972dc
Oct 29 22:18:16.138279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:16.138241    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7fkbf" event={"ID":"c2d52689-4162-48ee-abeb-84e3f606f656","Type":"ContainerStarted","Data":"8578c23178c927d7afa6d5b24eb0a44487a317f81cab95fa40235d94e30972dc"}
Oct 29 22:18:19.156877 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:19.156801    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7fkbf" event={"ID":"c2d52689-4162-48ee-abeb-84e3f606f656","Type":"ContainerStarted","Data":"2c4eeeb008f72901e427422b6c2d4d99a9c95cbd74af72bc5ab182de56046c97"}
Oct 29 22:18:19.195688 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:19.195631    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7fkbf" podStartSLOduration=2.13393954 podStartE2EDuration="5.195617214s" totalImagesPullingTime="3.061677674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:18:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:18:19.194784456 +0000 UTC m=+238.106228011" watchObservedRunningTime="2026-10-29 22:18:19.195617214 +0000 UTC m=+238.107060767"
Oct 29 22:18:24.906925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:24.906889    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-ggpjf"]
Oct 29 22:18:24.910202 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:24.910186    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-ggpjf"
Oct 29 22:18:24.924985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:24.924960    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9716fa1-328e-4c50-a811-e9bfef11ac64-tmp\") pod \"openshift-lws-operator-8465994787-ggpjf\" (UID: \"c9716fa1-328e-4c50-a811-e9bfef11ac64\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-ggpjf"
Oct 29 22:18:24.925125 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:24.925007    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcqtl\" (UniqueName: \"kubernetes.io/projected/c9716fa1-328e-4c50-a811-e9bfef11ac64-kube-api-access-zcqtl\") pod \"openshift-lws-operator-8465994787-ggpjf\" (UID: \"c9716fa1-328e-4c50-a811-e9bfef11ac64\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-ggpjf"
Oct 29 22:18:24.937488 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:24.937460    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 29 22:18:24.937785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:24.937756    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:18:24.937895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:24.937826    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9gg2j\""
Oct 29 22:18:24.938765 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:24.938746    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-ggpjf"]
Oct 29 22:18:25.025870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:25.025832    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9716fa1-328e-4c50-a811-e9bfef11ac64-tmp\") pod \"openshift-lws-operator-8465994787-ggpjf\" (UID: \"c9716fa1-328e-4c50-a811-e9bfef11ac64\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-ggpjf"
Oct 29 22:18:25.026043 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:25.025894    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcqtl\" (UniqueName: \"kubernetes.io/projected/c9716fa1-328e-4c50-a811-e9bfef11ac64-kube-api-access-zcqtl\") pod \"openshift-lws-operator-8465994787-ggpjf\" (UID: \"c9716fa1-328e-4c50-a811-e9bfef11ac64\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-ggpjf"
Oct 29 22:18:25.026183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:25.026162    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9716fa1-328e-4c50-a811-e9bfef11ac64-tmp\") pod \"openshift-lws-operator-8465994787-ggpjf\" (UID: \"c9716fa1-328e-4c50-a811-e9bfef11ac64\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-ggpjf"
Oct 29 22:18:25.045676 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:25.045647    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcqtl\" (UniqueName: \"kubernetes.io/projected/c9716fa1-328e-4c50-a811-e9bfef11ac64-kube-api-access-zcqtl\") pod \"openshift-lws-operator-8465994787-ggpjf\" (UID: \"c9716fa1-328e-4c50-a811-e9bfef11ac64\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-ggpjf"
Oct 29 22:18:25.221304 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:25.221209    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-ggpjf"
Oct 29 22:18:25.518126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:25.517926    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-ggpjf"]
Oct 29 22:18:25.520760 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:18:25.520734    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9716fa1_328e_4c50_a811_e9bfef11ac64.slice/crio-f215f17697ae05fcd30560ade300b2d9bbc2e22076dbeae8349624bab489b4ea WatchSource:0}: Error finding container f215f17697ae05fcd30560ade300b2d9bbc2e22076dbeae8349624bab489b4ea: Status 404 returned error can't find the container with id f215f17697ae05fcd30560ade300b2d9bbc2e22076dbeae8349624bab489b4ea
Oct 29 22:18:26.184422 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:26.184353    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-ggpjf" event={"ID":"c9716fa1-328e-4c50-a811-e9bfef11ac64","Type":"ContainerStarted","Data":"f215f17697ae05fcd30560ade300b2d9bbc2e22076dbeae8349624bab489b4ea"}
Oct 29 22:18:28.199636 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:28.199598    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-ggpjf" event={"ID":"c9716fa1-328e-4c50-a811-e9bfef11ac64","Type":"ContainerStarted","Data":"6f921bb7859fc23ae5d92750eda484fad9b83078d7b09068ffe1f8222c2709b1"}
Oct 29 22:18:28.238061 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:28.238007    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-ggpjf" podStartSLOduration=1.923021605 podStartE2EDuration="4.237991898s" totalImagesPullingTime="2.314970293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:18:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:18:28.237326517 +0000 UTC m=+247.148770071" watchObservedRunningTime="2026-10-29 22:18:28.237991898 +0000 UTC m=+247.149435452"
Oct 29 22:18:43.691680 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:18:43.691643    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:18:44.013868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.013783    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"]
Oct 29 22:18:44.017228 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.017210    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.057513 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.057474    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"]
Oct 29 22:18:44.075163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.075128    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d711fed7-3552-41ff-bca7-b5e79c526258-tmp\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.075163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.075165    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d711fed7-3552-41ff-bca7-b5e79c526258-manifests\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.075361 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.075187    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d711fed7-3552-41ff-bca7-b5e79c526258-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.075361 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.075308    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d711fed7-3552-41ff-bca7-b5e79c526258-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.075361 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.075352    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d711fed7-3552-41ff-bca7-b5e79c526258-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.075457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.075369    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ksf2\" (UniqueName: \"kubernetes.io/projected/d711fed7-3552-41ff-bca7-b5e79c526258-kube-api-access-8ksf2\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.112884 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.112771    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 29 22:18:44.113109 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.112947    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 29 22:18:44.113109 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.113090    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 29 22:18:44.113309 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.113290    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9jtb2\""
Oct 29 22:18:44.113783 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.113748    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 29 22:18:44.113900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.113891    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 29 22:18:44.176804 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.176765    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d711fed7-3552-41ff-bca7-b5e79c526258-manifests\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.176804 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.176801    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d711fed7-3552-41ff-bca7-b5e79c526258-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.177078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.176846    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d711fed7-3552-41ff-bca7-b5e79c526258-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.177078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.176879    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d711fed7-3552-41ff-bca7-b5e79c526258-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.177078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.176898    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ksf2\" (UniqueName: \"kubernetes.io/projected/d711fed7-3552-41ff-bca7-b5e79c526258-kube-api-access-8ksf2\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.177078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.177065    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d711fed7-3552-41ff-bca7-b5e79c526258-tmp\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.177335 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.177311    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d711fed7-3552-41ff-bca7-b5e79c526258-manifests\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.179464 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.179435    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d711fed7-3552-41ff-bca7-b5e79c526258-tmp\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.179604 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.179550    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d711fed7-3552-41ff-bca7-b5e79c526258-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.179680 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.179661    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d711fed7-3552-41ff-bca7-b5e79c526258-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.179847 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.179797    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d711fed7-3552-41ff-bca7-b5e79c526258-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.202118 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.202087    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ksf2\" (UniqueName: \"kubernetes.io/projected/d711fed7-3552-41ff-bca7-b5e79c526258-kube-api-access-8ksf2\") pod \"opendatahub-operator-controller-manager-6f7c578db4-jdgg4\" (UID: \"d711fed7-3552-41ff-bca7-b5e79c526258\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.332286 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.332197    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:44.680984 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:44.680961    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"]
Oct 29 22:18:44.682859 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:18:44.682835    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd711fed7_3552_41ff_bca7_b5e79c526258.slice/crio-185c0c28a69a0d42a465d52ae1416262183b77293c44e04a976677eb108d597d WatchSource:0}: Error finding container 185c0c28a69a0d42a465d52ae1416262183b77293c44e04a976677eb108d597d: Status 404 returned error can't find the container with id 185c0c28a69a0d42a465d52ae1416262183b77293c44e04a976677eb108d597d
Oct 29 22:18:45.262656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:45.262618    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4" event={"ID":"d711fed7-3552-41ff-bca7-b5e79c526258","Type":"ContainerStarted","Data":"185c0c28a69a0d42a465d52ae1416262183b77293c44e04a976677eb108d597d"}
Oct 29 22:18:48.275551 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:48.275469    4559 generic.go:356] "Generic (PLEG): container finished" podID="d711fed7-3552-41ff-bca7-b5e79c526258" containerID="fdfec2ffc9c767df6b229bb4a5e5ac9927ebb50ba17d4e27b489cb6a62c2aa3d" exitCode=0
Oct 29 22:18:48.275551 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:48.275528    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4" event={"ID":"d711fed7-3552-41ff-bca7-b5e79c526258","Type":"ContainerDied","Data":"fdfec2ffc9c767df6b229bb4a5e5ac9927ebb50ba17d4e27b489cb6a62c2aa3d"}
Oct 29 22:18:49.281547 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:49.281506    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4" event={"ID":"d711fed7-3552-41ff-bca7-b5e79c526258","Type":"ContainerStarted","Data":"73775ec5faf08ebc1473e4cc9cf9e961c2ff6e5f6b5df6709f7ed3e279da9e2e"}
Oct 29 22:18:49.282051 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:49.281628    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:18:49.364942 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:18:49.364890    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4" podStartSLOduration=2.736390626 podStartE2EDuration="6.364873933s" totalImagesPullingTime="3.211527037s" totalInitContainerRuntime="416.95627ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:18:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:18:49.351888845 +0000 UTC m=+268.263332399" watchObservedRunningTime="2026-10-29 22:18:49.364873933 +0000 UTC m=+268.276317483"
Oct 29 22:19:00.287789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:00.287759    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4"
Oct 29 22:19:03.369555 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.369524    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7b84b87b7c-gk9x6"]
Oct 29 22:19:03.372824 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.372794    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-gk9x6"
Oct 29 22:19:03.387445 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.387416    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 29 22:19:03.401167 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.401141    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b84b87b7c-gk9x6"]
Oct 29 22:19:03.466217 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.466161    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 29 22:19:03.466217 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.466199    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 29 22:19:03.466217 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.466221    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-tjdnx\""
Oct 29 22:19:03.487953 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.487926    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 29 22:19:03.551303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.551265    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8839f88-8053-4de1-b2d6-ecfbc851bb9c-tls-certs\") pod \"kube-auth-proxy-7b84b87b7c-gk9x6\" (UID: \"b8839f88-8053-4de1-b2d6-ecfbc851bb9c\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-gk9x6"
Oct 29 22:19:03.551472 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.551385    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmrc2\" (UniqueName: \"kubernetes.io/projected/b8839f88-8053-4de1-b2d6-ecfbc851bb9c-kube-api-access-dmrc2\") pod \"kube-auth-proxy-7b84b87b7c-gk9x6\" (UID: \"b8839f88-8053-4de1-b2d6-ecfbc851bb9c\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-gk9x6"
Oct 29 22:19:03.551472 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.551444    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8839f88-8053-4de1-b2d6-ecfbc851bb9c-tmp\") pod \"kube-auth-proxy-7b84b87b7c-gk9x6\" (UID: \"b8839f88-8053-4de1-b2d6-ecfbc851bb9c\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-gk9x6"
Oct 29 22:19:03.652237 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.652201    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmrc2\" (UniqueName: \"kubernetes.io/projected/b8839f88-8053-4de1-b2d6-ecfbc851bb9c-kube-api-access-dmrc2\") pod \"kube-auth-proxy-7b84b87b7c-gk9x6\" (UID: \"b8839f88-8053-4de1-b2d6-ecfbc851bb9c\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-gk9x6"
Oct 29 22:19:03.652416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.652260    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8839f88-8053-4de1-b2d6-ecfbc851bb9c-tmp\") pod \"kube-auth-proxy-7b84b87b7c-gk9x6\" (UID: \"b8839f88-8053-4de1-b2d6-ecfbc851bb9c\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-gk9x6"
Oct 29 22:19:03.652416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.652284    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8839f88-8053-4de1-b2d6-ecfbc851bb9c-tls-certs\") pod \"kube-auth-proxy-7b84b87b7c-gk9x6\" (UID: \"b8839f88-8053-4de1-b2d6-ecfbc851bb9c\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-gk9x6"
Oct 29 22:19:03.654639 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.654609    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8839f88-8053-4de1-b2d6-ecfbc851bb9c-tmp\") pod \"kube-auth-proxy-7b84b87b7c-gk9x6\" (UID: \"b8839f88-8053-4de1-b2d6-ecfbc851bb9c\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-gk9x6"
Oct 29 22:19:03.654909 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.654890    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8839f88-8053-4de1-b2d6-ecfbc851bb9c-tls-certs\") pod \"kube-auth-proxy-7b84b87b7c-gk9x6\" (UID: \"b8839f88-8053-4de1-b2d6-ecfbc851bb9c\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-gk9x6"
Oct 29 22:19:03.671692 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.671663    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmrc2\" (UniqueName: \"kubernetes.io/projected/b8839f88-8053-4de1-b2d6-ecfbc851bb9c-kube-api-access-dmrc2\") pod \"kube-auth-proxy-7b84b87b7c-gk9x6\" (UID: \"b8839f88-8053-4de1-b2d6-ecfbc851bb9c\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-gk9x6"
Oct 29 22:19:03.685543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.685514    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-gk9x6"
Oct 29 22:19:03.984300 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:03.984272    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b84b87b7c-gk9x6"]
Oct 29 22:19:03.987165 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:19:03.987137    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8839f88_8053_4de1_b2d6_ecfbc851bb9c.slice/crio-bed9d01b46e24a9c0f3ddce24e8cafa395d5e72b053fa49a365ebad30dacbe65 WatchSource:0}: Error finding container bed9d01b46e24a9c0f3ddce24e8cafa395d5e72b053fa49a365ebad30dacbe65: Status 404 returned error can't find the container with id bed9d01b46e24a9c0f3ddce24e8cafa395d5e72b053fa49a365ebad30dacbe65
Oct 29 22:19:04.338702 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.338617    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-gk9x6" event={"ID":"b8839f88-8053-4de1-b2d6-ecfbc851bb9c","Type":"ContainerStarted","Data":"bed9d01b46e24a9c0f3ddce24e8cafa395d5e72b053fa49a365ebad30dacbe65"}
Oct 29 22:19:04.618575 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.618499    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"]
Oct 29 22:19:04.621977 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.621948    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.638679 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.638648    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 29 22:19:04.638891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.638871    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 29 22:19:04.638986 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.638875    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 29 22:19:04.650584 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.650553    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 29 22:19:04.650584 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.650568    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rgwvk\""
Oct 29 22:19:04.655002 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.654977    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"]
Oct 29 22:19:04.764254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.764214    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.764456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.764260    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.764456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.764291    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvpdl\" (UniqueName: \"kubernetes.io/projected/0f1b3de4-c472-4d23-8368-31bbe464c148-kube-api-access-cvpdl\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.764573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.764483    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.764573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.764541    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1b3de4-c472-4d23-8368-31bbe464c148-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.764673 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.764640    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.764730 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.764710    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.865260 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.865219    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1b3de4-c472-4d23-8368-31bbe464c148-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.865440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.865283    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.865440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.865334    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.865440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.865375    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.865440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.865397    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.865440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.865422    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvpdl\" (UniqueName: \"kubernetes.io/projected/0f1b3de4-c472-4d23-8368-31bbe464c148-kube-api-access-cvpdl\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.865687 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.865474    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.866516 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.866206    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.868446 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.868419    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1b3de4-c472-4d23-8368-31bbe464c148-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.868956 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.868900    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.869053 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.869005    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.869103 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.869054    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.886672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.886609    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.887650 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.887623    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvpdl\" (UniqueName: \"kubernetes.io/projected/0f1b3de4-c472-4d23-8368-31bbe464c148-kube-api-access-cvpdl\") pod \"istiod-openshift-gateway-7876bfcb74-szfl8\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:04.937459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:04.937421    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:05.269320 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:05.269286    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"]
Oct 29 22:19:05.272178 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:19:05.272148    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1b3de4_c472_4d23_8368_31bbe464c148.slice/crio-07c78b0167323242393b9bb8c1046ef7e281f6d2f9b365fd5891be8e53a04118 WatchSource:0}: Error finding container 07c78b0167323242393b9bb8c1046ef7e281f6d2f9b365fd5891be8e53a04118: Status 404 returned error can't find the container with id 07c78b0167323242393b9bb8c1046ef7e281f6d2f9b365fd5891be8e53a04118
Oct 29 22:19:05.345275 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:05.345242    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8" event={"ID":"0f1b3de4-c472-4d23-8368-31bbe464c148","Type":"ContainerStarted","Data":"07c78b0167323242393b9bb8c1046ef7e281f6d2f9b365fd5891be8e53a04118"}
Oct 29 22:19:07.356754 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:07.356720    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-gk9x6" event={"ID":"b8839f88-8053-4de1-b2d6-ecfbc851bb9c","Type":"ContainerStarted","Data":"aef0c5e63258f36c8047cf48fad1d4beb70374d14dcd0ac9fe623b2dd471843e"}
Oct 29 22:19:07.396287 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:07.396229    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-gk9x6" podStartSLOduration=1.829314989 podStartE2EDuration="4.396208239s" totalImagesPullingTime="2.56689325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:07.394866342 +0000 UTC m=+286.306309894" watchObservedRunningTime="2026-10-29 22:19:07.396208239 +0000 UTC m=+286.307651794"
Oct 29 22:19:07.734614 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:07.734585    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:19:07.734676 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:07.734648    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:19:08.362419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:08.362378    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8" event={"ID":"0f1b3de4-c472-4d23-8368-31bbe464c148","Type":"ContainerStarted","Data":"25ce15327d5dcc4de8c7aa55256dc958cfa7dce4a432cf904f596e3f1eb87baa"}
Oct 29 22:19:08.362911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:08.362574    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:08.364204 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:08.364170    4559 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-szfl8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:19:08.364315 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:08.364227    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8" podUID="0f1b3de4-c472-4d23-8368-31bbe464c148" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:19:08.417359 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:08.417293    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8" podStartSLOduration=1.957935916 podStartE2EDuration="4.417273513s" totalImagesPullingTime="2.459337597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:08.414092617 +0000 UTC m=+287.325536170" watchObservedRunningTime="2026-10-29 22:19:08.417273513 +0000 UTC m=+287.328717068"
Oct 29 22:19:09.366883 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:09.366857    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:19:13.693067 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:19:13.693033    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:19:21.248978 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:21.248950    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:19:21.249360 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:19:21.249331    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:19:43.692646 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:19:43.692560    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:20:05.558883 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.558846    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"]
Oct 29 22:20:05.562129 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.562107    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.592175 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.592139    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"]
Oct 29 22:20:05.594744 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.594715    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj9x6\" (UniqueName: \"kubernetes.io/projected/f06f3791-4893-4cec-a5f3-9c0004bc2842-kube-api-access-fj9x6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.594895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.594757    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f06f3791-4893-4cec-a5f3-9c0004bc2842-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.594895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.594786    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f06f3791-4893-4cec-a5f3-9c0004bc2842-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.594973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.594950    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06f3791-4893-4cec-a5f3-9c0004bc2842-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.595014 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.594987    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06f3791-4893-4cec-a5f3-9c0004bc2842-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.595118 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.595100    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f06f3791-4893-4cec-a5f3-9c0004bc2842-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.595181 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.595166    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f06f3791-4893-4cec-a5f3-9c0004bc2842-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.695843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.695770    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f06f3791-4893-4cec-a5f3-9c0004bc2842-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.696053 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.695865    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj9x6\" (UniqueName: \"kubernetes.io/projected/f06f3791-4893-4cec-a5f3-9c0004bc2842-kube-api-access-fj9x6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.696053 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.695896    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f06f3791-4893-4cec-a5f3-9c0004bc2842-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.696053 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.695922    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f06f3791-4893-4cec-a5f3-9c0004bc2842-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.696053 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.695960    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06f3791-4893-4cec-a5f3-9c0004bc2842-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.696053 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.695980    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06f3791-4893-4cec-a5f3-9c0004bc2842-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.696053 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.696045    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f06f3791-4893-4cec-a5f3-9c0004bc2842-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.696709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.696683    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f06f3791-4893-4cec-a5f3-9c0004bc2842-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.698596 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.698571    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f06f3791-4893-4cec-a5f3-9c0004bc2842-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.698699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.698571    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f06f3791-4893-4cec-a5f3-9c0004bc2842-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.698699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.698677    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06f3791-4893-4cec-a5f3-9c0004bc2842-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.698832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.698794    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f06f3791-4893-4cec-a5f3-9c0004bc2842-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.714782 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.714749    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06f3791-4893-4cec-a5f3-9c0004bc2842-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.714926 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.714755    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj9x6\" (UniqueName: \"kubernetes.io/projected/f06f3791-4893-4cec-a5f3-9c0004bc2842-kube-api-access-fj9x6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhpfs\" (UID: \"f06f3791-4893-4cec-a5f3-9c0004bc2842\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:05.873463 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:05.873377    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:06.186999 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:06.186969    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"]
Oct 29 22:20:06.189068 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:20:06.189038    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06f3791_4893_4cec_a5f3_9c0004bc2842.slice/crio-f2ae8ebc520c001951527cd7ed9490ba507163fb8d52fcc2e61308e018da1626 WatchSource:0}: Error finding container f2ae8ebc520c001951527cd7ed9490ba507163fb8d52fcc2e61308e018da1626: Status 404 returned error can't find the container with id f2ae8ebc520c001951527cd7ed9490ba507163fb8d52fcc2e61308e018da1626
Oct 29 22:20:06.191181 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:06.191146    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:20:06.191282 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:06.191224    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:20:06.594735 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:06.594648    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs" event={"ID":"f06f3791-4893-4cec-a5f3-9c0004bc2842","Type":"ContainerStarted","Data":"06a05b6cd7da2a75774e2d0266db299456aa63c1b0de031257dfcd5de7d47476"}
Oct 29 22:20:06.594735 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:06.594698    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs" event={"ID":"f06f3791-4893-4cec-a5f3-9c0004bc2842","Type":"ContainerStarted","Data":"f2ae8ebc520c001951527cd7ed9490ba507163fb8d52fcc2e61308e018da1626"}
Oct 29 22:20:06.595194 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:06.594792    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:07.600623 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:07.600594    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs"
Oct 29 22:20:07.640672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:07.640615    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhpfs" podStartSLOduration=2.640600411 podStartE2EDuration="2.640600411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:06.658727367 +0000 UTC m=+345.570170922" watchObservedRunningTime="2026-10-29 22:20:07.640600411 +0000 UTC m=+346.552043965"
Oct 29 22:20:07.661660 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:07.661628    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"]
Oct 29 22:20:07.661927 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:07.661899    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8" podUID="0f1b3de4-c472-4d23-8368-31bbe464c148" containerName="discovery" containerID="cri-o://25ce15327d5dcc4de8c7aa55256dc958cfa7dce4a432cf904f596e3f1eb87baa" gracePeriod=30
Oct 29 22:20:07.923630 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:07.923600    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:20:08.018701 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.018664    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f1b3de4-c472-4d23-8368-31bbe464c148-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1b3de4-c472-4d23-8368-31bbe464c148-local-certs\") pod \"0f1b3de4-c472-4d23-8368-31bbe464c148\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") "
Oct 29 22:20:08.018911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.018713    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-csr-ca-configmap\") pod \"0f1b3de4-c472-4d23-8368-31bbe464c148\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") "
Oct 29 22:20:08.018911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.018774    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-token\" (UniqueName: \"kubernetes.io/projected/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-token\") pod \"0f1b3de4-c472-4d23-8368-31bbe464c148\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") "
Oct 29 22:20:08.018911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.018834    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-kubeconfig\") pod \"0f1b3de4-c472-4d23-8368-31bbe464c148\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") "
Oct 29 22:20:08.018911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.018884    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f1b3de4-c472-4d23-8368-31bbe464c148-kube-api-access-cvpdl\" (UniqueName: \"kubernetes.io/projected/0f1b3de4-c472-4d23-8368-31bbe464c148-kube-api-access-cvpdl\") pod \"0f1b3de4-c472-4d23-8368-31bbe464c148\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") "
Oct 29 22:20:08.019119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.018935    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-cacerts\" (UniqueName: \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-cacerts\") pod \"0f1b3de4-c472-4d23-8368-31bbe464c148\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") "
Oct 29 22:20:08.019119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.019019    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-csr-dns-cert\") pod \"0f1b3de4-c472-4d23-8368-31bbe464c148\" (UID: \"0f1b3de4-c472-4d23-8368-31bbe464c148\") "
Oct 29 22:20:08.019339 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.019310    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-csr-ca-configmap" pod "0f1b3de4-c472-4d23-8368-31bbe464c148" (UID: "0f1b3de4-c472-4d23-8368-31bbe464c148"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:20:08.021424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.021390    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-token" pod "0f1b3de4-c472-4d23-8368-31bbe464c148" (UID: "0f1b3de4-c472-4d23-8368-31bbe464c148"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:20:08.021562 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.021456    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1b3de4-c472-4d23-8368-31bbe464c148-kube-api-access-cvpdl" pod "0f1b3de4-c472-4d23-8368-31bbe464c148" (UID: "0f1b3de4-c472-4d23-8368-31bbe464c148"). InnerVolumeSpecName "kube-api-access-cvpdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:20:08.021699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.021673    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1b3de4-c472-4d23-8368-31bbe464c148-local-certs" pod "0f1b3de4-c472-4d23-8368-31bbe464c148" (UID: "0f1b3de4-c472-4d23-8368-31bbe464c148"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:20:08.021699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.021670    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-kubeconfig" pod "0f1b3de4-c472-4d23-8368-31bbe464c148" (UID: "0f1b3de4-c472-4d23-8368-31bbe464c148"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:20:08.021885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.021862    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-csr-dns-cert" pod "0f1b3de4-c472-4d23-8368-31bbe464c148" (UID: "0f1b3de4-c472-4d23-8368-31bbe464c148"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:20:08.021930 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.021871    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-cacerts" pod "0f1b3de4-c472-4d23-8368-31bbe464c148" (UID: "0f1b3de4-c472-4d23-8368-31bbe464c148"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:20:08.120421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.120378    4559 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1b3de4-c472-4d23-8368-31bbe464c148-local-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:20:08.120421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.120414    4559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-csr-ca-configmap\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:20:08.120421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.120424    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:20:08.120421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.120432    4559 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-kubeconfig\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:20:08.120695 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.120441    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvpdl\" (UniqueName: \"kubernetes.io/projected/0f1b3de4-c472-4d23-8368-31bbe464c148-kube-api-access-cvpdl\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:20:08.120695 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.120448    4559 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-cacerts\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:20:08.120695 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.120459    4559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f1b3de4-c472-4d23-8368-31bbe464c148-istio-csr-dns-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:20:08.604683 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.604642    4559 generic.go:356] "Generic (PLEG): container finished" podID="0f1b3de4-c472-4d23-8368-31bbe464c148" containerID="25ce15327d5dcc4de8c7aa55256dc958cfa7dce4a432cf904f596e3f1eb87baa" exitCode=0
Oct 29 22:20:08.605119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.604697    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"
Oct 29 22:20:08.605119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.604729    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8" event={"ID":"0f1b3de4-c472-4d23-8368-31bbe464c148","Type":"ContainerDied","Data":"25ce15327d5dcc4de8c7aa55256dc958cfa7dce4a432cf904f596e3f1eb87baa"}
Oct 29 22:20:08.605119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.604794    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8" event={"ID":"0f1b3de4-c472-4d23-8368-31bbe464c148","Type":"ContainerDied","Data":"07c78b0167323242393b9bb8c1046ef7e281f6d2f9b365fd5891be8e53a04118"}
Oct 29 22:20:08.605119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.604830    4559 scope.go:122] "RemoveContainer" containerID="25ce15327d5dcc4de8c7aa55256dc958cfa7dce4a432cf904f596e3f1eb87baa"
Oct 29 22:20:08.613470 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.613452    4559 scope.go:122] "RemoveContainer" containerID="25ce15327d5dcc4de8c7aa55256dc958cfa7dce4a432cf904f596e3f1eb87baa"
Oct 29 22:20:08.613784 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:20:08.613766    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ce15327d5dcc4de8c7aa55256dc958cfa7dce4a432cf904f596e3f1eb87baa\": container with ID starting with 25ce15327d5dcc4de8c7aa55256dc958cfa7dce4a432cf904f596e3f1eb87baa not found: ID does not exist" containerID="25ce15327d5dcc4de8c7aa55256dc958cfa7dce4a432cf904f596e3f1eb87baa"
Oct 29 22:20:08.613849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.613794    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ce15327d5dcc4de8c7aa55256dc958cfa7dce4a432cf904f596e3f1eb87baa"} err="failed to get container status \"25ce15327d5dcc4de8c7aa55256dc958cfa7dce4a432cf904f596e3f1eb87baa\": rpc error: code = NotFound desc = could not find container \"25ce15327d5dcc4de8c7aa55256dc958cfa7dce4a432cf904f596e3f1eb87baa\": container with ID starting with 25ce15327d5dcc4de8c7aa55256dc958cfa7dce4a432cf904f596e3f1eb87baa not found: ID does not exist"
Oct 29 22:20:08.666396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.666360    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"]
Oct 29 22:20:08.678718 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:08.678675    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-szfl8"]
Oct 29 22:20:09.226122 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:20:09.226082    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f1b3de4-c472-4d23-8368-31bbe464c148" path="/var/lib/kubelet/pods/0f1b3de4-c472-4d23-8368-31bbe464c148/volumes"
Oct 29 22:20:13.692490 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:20:13.692446    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:20:43.692187 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:20:43.692147    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:21:13.694665 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:21:13.694573    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:21:26.067565 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:26.067530    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-th4wc"]
Oct 29 22:21:26.068050 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:26.067882    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0f1b3de4-c472-4d23-8368-31bbe464c148" containerName="discovery"
Oct 29 22:21:26.068050 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:26.067893    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f1b3de4-c472-4d23-8368-31bbe464c148" containerName="discovery"
Oct 29 22:21:26.068050 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:26.067964    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f1b3de4-c472-4d23-8368-31bbe464c148" containerName="discovery"
Oct 29 22:21:26.071403 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:26.071379    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-th4wc"
Oct 29 22:21:26.089866 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:26.089836    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-7l6js\""
Oct 29 22:21:26.101994 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:26.101962    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 29 22:21:26.102170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:26.102035    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 29 22:21:26.103462 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:26.103440    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-th4wc"]
Oct 29 22:21:26.199654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:26.199611    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtmpp\" (UniqueName: \"kubernetes.io/projected/33b89a91-a78b-42e0-ac39-3d25c83d8844-kube-api-access-qtmpp\") pod \"authorino-operator-68d6dd76dc-th4wc\" (UID: \"33b89a91-a78b-42e0-ac39-3d25c83d8844\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-th4wc"
Oct 29 22:21:26.300505 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:26.300468    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtmpp\" (UniqueName: \"kubernetes.io/projected/33b89a91-a78b-42e0-ac39-3d25c83d8844-kube-api-access-qtmpp\") pod \"authorino-operator-68d6dd76dc-th4wc\" (UID: \"33b89a91-a78b-42e0-ac39-3d25c83d8844\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-th4wc"
Oct 29 22:21:26.322285 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:26.322212    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtmpp\" (UniqueName: \"kubernetes.io/projected/33b89a91-a78b-42e0-ac39-3d25c83d8844-kube-api-access-qtmpp\") pod \"authorino-operator-68d6dd76dc-th4wc\" (UID: \"33b89a91-a78b-42e0-ac39-3d25c83d8844\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-th4wc"
Oct 29 22:21:26.383871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:26.383831    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-th4wc"
Oct 29 22:21:26.678435 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:26.678398    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-th4wc"]
Oct 29 22:21:26.683786 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:21:26.683753    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b89a91_a78b_42e0_ac39_3d25c83d8844.slice/crio-7e1e5db86bf389cc80970b9ce9107dde15f070ad13b33d586ae7a8c1e3114c50 WatchSource:0}: Error finding container 7e1e5db86bf389cc80970b9ce9107dde15f070ad13b33d586ae7a8c1e3114c50: Status 404 returned error can't find the container with id 7e1e5db86bf389cc80970b9ce9107dde15f070ad13b33d586ae7a8c1e3114c50
Oct 29 22:21:26.685841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:26.685821    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:21:26.930031 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:26.929953    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-th4wc" event={"ID":"33b89a91-a78b-42e0-ac39-3d25c83d8844","Type":"ContainerStarted","Data":"7e1e5db86bf389cc80970b9ce9107dde15f070ad13b33d586ae7a8c1e3114c50"}
Oct 29 22:21:28.357670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:28.357628    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-f92wp"]
Oct 29 22:21:28.366594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:28.366567    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f92wp"
Oct 29 22:21:28.386610 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:28.386581    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 29 22:21:28.394684 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:28.394654    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-rtxf2\""
Oct 29 22:21:28.400552 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:28.400518    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-f92wp"]
Oct 29 22:21:28.421000 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:28.420964    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdw2t\" (UniqueName: \"kubernetes.io/projected/1ded2baf-62e1-43f1-8489-06e74ea61f5f-kube-api-access-tdw2t\") pod \"dns-operator-controller-manager-65bfd79598-f92wp\" (UID: \"1ded2baf-62e1-43f1-8489-06e74ea61f5f\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f92wp"
Oct 29 22:21:28.521859 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:28.521821    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdw2t\" (UniqueName: \"kubernetes.io/projected/1ded2baf-62e1-43f1-8489-06e74ea61f5f-kube-api-access-tdw2t\") pod \"dns-operator-controller-manager-65bfd79598-f92wp\" (UID: \"1ded2baf-62e1-43f1-8489-06e74ea61f5f\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f92wp"
Oct 29 22:21:28.544071 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:28.544032    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdw2t\" (UniqueName: \"kubernetes.io/projected/1ded2baf-62e1-43f1-8489-06e74ea61f5f-kube-api-access-tdw2t\") pod \"dns-operator-controller-manager-65bfd79598-f92wp\" (UID: \"1ded2baf-62e1-43f1-8489-06e74ea61f5f\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f92wp"
Oct 29 22:21:28.682579 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:28.682538    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f92wp"
Oct 29 22:21:28.994368 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:28.994347    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-f92wp"]
Oct 29 22:21:28.996254 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:21:28.996227    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ded2baf_62e1_43f1_8489_06e74ea61f5f.slice/crio-7a26d7ac3633a81f185a92f0e879134cd57bc737f3ba5d2120462e5133fb8f12 WatchSource:0}: Error finding container 7a26d7ac3633a81f185a92f0e879134cd57bc737f3ba5d2120462e5133fb8f12: Status 404 returned error can't find the container with id 7a26d7ac3633a81f185a92f0e879134cd57bc737f3ba5d2120462e5133fb8f12
Oct 29 22:21:29.949748 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:29.949706    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f92wp" event={"ID":"1ded2baf-62e1-43f1-8489-06e74ea61f5f","Type":"ContainerStarted","Data":"7a26d7ac3633a81f185a92f0e879134cd57bc737f3ba5d2120462e5133fb8f12"}
Oct 29 22:21:29.951704 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:29.951671    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-th4wc" event={"ID":"33b89a91-a78b-42e0-ac39-3d25c83d8844","Type":"ContainerStarted","Data":"bc5e527d7163593bfd2b90563ecbe1cb3eae6ca9cd48c0f0bd115eb0e127dc9d"}
Oct 29 22:21:29.951934 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:29.951916    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-th4wc"
Oct 29 22:21:29.994769 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:29.994715    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-th4wc" podStartSLOduration=1.73545156 podStartE2EDuration="3.994700629s" totalImagesPullingTime="2.259249069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:21:29.994450757 +0000 UTC m=+428.905894323" watchObservedRunningTime="2026-10-29 22:21:29.994700629 +0000 UTC m=+428.906144182"
Oct 29 22:21:30.783975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:30.783932    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"]
Oct 29 22:21:30.787396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:30.787372    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"
Oct 29 22:21:30.805756 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:30.805734    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-mvb4w\""
Oct 29 22:21:30.819878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:30.819803    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"]
Oct 29 22:21:30.843012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:30.842970    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kwlk\" (UniqueName: \"kubernetes.io/projected/11ea846f-9175-4f0b-9e75-d6647682bda1-kube-api-access-8kwlk\") pod \"limitador-operator-controller-manager-856dd584c5-qcwsw\" (UID: \"11ea846f-9175-4f0b-9e75-d6647682bda1\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"
Oct 29 22:21:30.943667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:30.943621    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kwlk\" (UniqueName: \"kubernetes.io/projected/11ea846f-9175-4f0b-9e75-d6647682bda1-kube-api-access-8kwlk\") pod \"limitador-operator-controller-manager-856dd584c5-qcwsw\" (UID: \"11ea846f-9175-4f0b-9e75-d6647682bda1\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"
Oct 29 22:21:30.957895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:30.957848    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f92wp" event={"ID":"1ded2baf-62e1-43f1-8489-06e74ea61f5f","Type":"ContainerStarted","Data":"443aa51a8679ac6ed18bace151396e63e7db7c47c5c814af5b668c4e3ba2e09e"}
Oct 29 22:21:30.958302 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:30.958004    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f92wp"
Oct 29 22:21:30.963491 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:30.963464    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kwlk\" (UniqueName: \"kubernetes.io/projected/11ea846f-9175-4f0b-9e75-d6647682bda1-kube-api-access-8kwlk\") pod \"limitador-operator-controller-manager-856dd584c5-qcwsw\" (UID: \"11ea846f-9175-4f0b-9e75-d6647682bda1\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"
Oct 29 22:21:31.001947 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:31.001892    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f92wp" podStartSLOduration=1.885838388 podStartE2EDuration="3.001877158s" totalImagesPullingTime="1.11603877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:21:31.000597293 +0000 UTC m=+429.912040872" watchObservedRunningTime="2026-10-29 22:21:31.001877158 +0000 UTC m=+429.913320733"
Oct 29 22:21:31.100575 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:31.100487    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"
Oct 29 22:21:31.440526 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:31.440501    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"]
Oct 29 22:21:31.442881 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:21:31.442851    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ea846f_9175_4f0b_9e75_d6647682bda1.slice/crio-4dffb8fb17cfcad01b22f14334692d04b04c1699f78197c901db28938e4aeb7d WatchSource:0}: Error finding container 4dffb8fb17cfcad01b22f14334692d04b04c1699f78197c901db28938e4aeb7d: Status 404 returned error can't find the container with id 4dffb8fb17cfcad01b22f14334692d04b04c1699f78197c901db28938e4aeb7d
Oct 29 22:21:31.966543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:31.966493    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw" event={"ID":"11ea846f-9175-4f0b-9e75-d6647682bda1","Type":"ContainerStarted","Data":"4dffb8fb17cfcad01b22f14334692d04b04c1699f78197c901db28938e4aeb7d"}
Oct 29 22:21:32.971364 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:32.971322    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw" event={"ID":"11ea846f-9175-4f0b-9e75-d6647682bda1","Type":"ContainerStarted","Data":"e98da5dce02aa44e4aef57ce03f58b9428d8709432cf9de4c25ea991b92fa4c7"}
Oct 29 22:21:32.971882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:32.971468    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"
Oct 29 22:21:33.010378 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:33.010324    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw" podStartSLOduration=1.844349834 podStartE2EDuration="3.010309656s" totalImagesPullingTime="1.165959822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:21:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:21:33.009490809 +0000 UTC m=+431.920934374" watchObservedRunningTime="2026-10-29 22:21:33.010309656 +0000 UTC m=+431.921753209"
Oct 29 22:21:34.942856 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:34.942803    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"]
Oct 29 22:21:34.946356 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:34.946332    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:34.964331 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:34.964300    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6hs2c\""
Oct 29 22:21:34.977899 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:34.977870    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"]
Oct 29 22:21:35.083345 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:35.083308    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tdx4\" (UniqueName: \"kubernetes.io/projected/88927588-2efd-4b81-81c5-a7ced3827020-kube-api-access-2tdx4\") pod \"kuadrant-operator-controller-manager-6b6fb56846-t87wd\" (UID: \"88927588-2efd-4b81-81c5-a7ced3827020\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:35.083536 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:35.083383    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88927588-2efd-4b81-81c5-a7ced3827020-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-t87wd\" (UID: \"88927588-2efd-4b81-81c5-a7ced3827020\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:35.184896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:35.184845    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tdx4\" (UniqueName: \"kubernetes.io/projected/88927588-2efd-4b81-81c5-a7ced3827020-kube-api-access-2tdx4\") pod \"kuadrant-operator-controller-manager-6b6fb56846-t87wd\" (UID: \"88927588-2efd-4b81-81c5-a7ced3827020\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:35.185081 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:35.184946    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88927588-2efd-4b81-81c5-a7ced3827020-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-t87wd\" (UID: \"88927588-2efd-4b81-81c5-a7ced3827020\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:35.185282 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:35.185266    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88927588-2efd-4b81-81c5-a7ced3827020-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-t87wd\" (UID: \"88927588-2efd-4b81-81c5-a7ced3827020\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:35.213726 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:35.213671    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tdx4\" (UniqueName: \"kubernetes.io/projected/88927588-2efd-4b81-81c5-a7ced3827020-kube-api-access-2tdx4\") pod \"kuadrant-operator-controller-manager-6b6fb56846-t87wd\" (UID: \"88927588-2efd-4b81-81c5-a7ced3827020\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:35.258690 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:35.258650    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:35.589493 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:35.589459    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"]
Oct 29 22:21:35.591526 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:21:35.591493    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88927588_2efd_4b81_81c5_a7ced3827020.slice/crio-60b928cc8577e1da73c9d6610129329fe2625ea85ea6a1afc78ba97561f2e946 WatchSource:0}: Error finding container 60b928cc8577e1da73c9d6610129329fe2625ea85ea6a1afc78ba97561f2e946: Status 404 returned error can't find the container with id 60b928cc8577e1da73c9d6610129329fe2625ea85ea6a1afc78ba97561f2e946
Oct 29 22:21:35.986595 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:35.986558    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd" event={"ID":"88927588-2efd-4b81-81c5-a7ced3827020","Type":"ContainerStarted","Data":"60b928cc8577e1da73c9d6610129329fe2625ea85ea6a1afc78ba97561f2e946"}
Oct 29 22:21:40.006445 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:40.006409    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd" event={"ID":"88927588-2efd-4b81-81c5-a7ced3827020","Type":"ContainerStarted","Data":"9b121d0ccd0a7a0989819ed1de95efd4a6f3c7bcacc911ef1b34d3379ed72a4b"}
Oct 29 22:21:40.006899 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:40.006561    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:40.071048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:40.070986    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd" podStartSLOduration=2.16916502 podStartE2EDuration="6.070967444s" totalImagesPullingTime="3.901802424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:21:40.058073965 +0000 UTC m=+438.969517519" watchObservedRunningTime="2026-10-29 22:21:40.070967444 +0000 UTC m=+438.982411000"
Oct 29 22:21:40.960253 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:40.960220    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-th4wc"
Oct 29 22:21:41.969461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:41.969426    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f92wp"
Oct 29 22:21:43.692015 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:21:43.691978    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:21:43.977835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:43.977732    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"
Oct 29 22:21:51.013010 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:51.012981    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:54.417054 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.417004    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"]
Oct 29 22:21:54.417507 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.417366    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd" podUID="88927588-2efd-4b81-81c5-a7ced3827020" containerName="manager" containerID="cri-o://9b121d0ccd0a7a0989819ed1de95efd4a6f3c7bcacc911ef1b34d3379ed72a4b" gracePeriod=2
Oct 29 22:21:54.418999 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.418975    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"]
Oct 29 22:21:54.431033 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:21:54.430974    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-t87wd\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="88927588-2efd-4b81-81c5-a7ced3827020" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:54.432133 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.432105    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"]
Oct 29 22:21:54.432735 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.432716    4559 cpu_manager.go:414] "containerMap: removing container" podUID="88927588-2efd-4b81-81c5-a7ced3827020" containerName="manager"
Oct 29 22:21:54.432849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.432737    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88927588-2efd-4b81-81c5-a7ced3827020" containerName="manager"
Oct 29 22:21:54.432849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.432834    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="88927588-2efd-4b81-81c5-a7ced3827020" containerName="manager"
Oct 29 22:21:54.436063 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.436043    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"
Oct 29 22:21:54.446689 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.446658    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"]
Oct 29 22:21:54.446915 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.446878    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw" podUID="11ea846f-9175-4f0b-9e75-d6647682bda1" containerName="manager" containerID="cri-o://e98da5dce02aa44e4aef57ce03f58b9428d8709432cf9de4c25ea991b92fa4c7" gracePeriod=2
Oct 29 22:21:54.452918 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.452891    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"]
Oct 29 22:21:54.453954 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.453934    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3758ef38-0907-46fe-b936-c3c62361f2bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-f92lz\" (UID: \"3758ef38-0907-46fe-b936-c3c62361f2bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"
Oct 29 22:21:54.454008 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.453963    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j6cv\" (UniqueName: \"kubernetes.io/projected/3758ef38-0907-46fe-b936-c3c62361f2bb-kube-api-access-7j6cv\") pod \"kuadrant-operator-controller-manager-6b6fb56846-f92lz\" (UID: \"3758ef38-0907-46fe-b936-c3c62361f2bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"
Oct 29 22:21:54.464401 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.464365    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"]
Oct 29 22:21:54.476386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.476348    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-q9l85"]
Oct 29 22:21:54.476948 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.476927    4559 cpu_manager.go:414] "containerMap: removing container" podUID="11ea846f-9175-4f0b-9e75-d6647682bda1" containerName="manager"
Oct 29 22:21:54.477037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.476951    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ea846f-9175-4f0b-9e75-d6647682bda1" containerName="manager"
Oct 29 22:21:54.477095 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.477063    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ea846f-9175-4f0b-9e75-d6647682bda1" containerName="manager"
Oct 29 22:21:54.483027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.483003    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-q9l85"
Oct 29 22:21:54.501160 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.501128    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-q9l85"]
Oct 29 22:21:54.508396 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:21:54.508360    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-t87wd\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="88927588-2efd-4b81-81c5-a7ced3827020" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:54.521640 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:21:54.521601    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-t87wd\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="88927588-2efd-4b81-81c5-a7ced3827020" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:54.554777 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.554747    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l78qc\" (UniqueName: \"kubernetes.io/projected/6754baae-fae7-40dd-ab9b-54fd68603755-kube-api-access-l78qc\") pod \"limitador-operator-controller-manager-856dd584c5-q9l85\" (UID: \"6754baae-fae7-40dd-ab9b-54fd68603755\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-q9l85"
Oct 29 22:21:54.554938 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.554790    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3758ef38-0907-46fe-b936-c3c62361f2bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-f92lz\" (UID: \"3758ef38-0907-46fe-b936-c3c62361f2bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"
Oct 29 22:21:54.555009 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.554978    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j6cv\" (UniqueName: \"kubernetes.io/projected/3758ef38-0907-46fe-b936-c3c62361f2bb-kube-api-access-7j6cv\") pod \"kuadrant-operator-controller-manager-6b6fb56846-f92lz\" (UID: \"3758ef38-0907-46fe-b936-c3c62361f2bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"
Oct 29 22:21:54.555202 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.555176    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3758ef38-0907-46fe-b936-c3c62361f2bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-f92lz\" (UID: \"3758ef38-0907-46fe-b936-c3c62361f2bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"
Oct 29 22:21:54.574256 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:21:54.574211    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-qcwsw\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="11ea846f-9175-4f0b-9e75-d6647682bda1" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"
Oct 29 22:21:54.574568 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.574549    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j6cv\" (UniqueName: \"kubernetes.io/projected/3758ef38-0907-46fe-b936-c3c62361f2bb-kube-api-access-7j6cv\") pod \"kuadrant-operator-controller-manager-6b6fb56846-f92lz\" (UID: \"3758ef38-0907-46fe-b936-c3c62361f2bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"
Oct 29 22:21:54.656067 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.656033    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l78qc\" (UniqueName: \"kubernetes.io/projected/6754baae-fae7-40dd-ab9b-54fd68603755-kube-api-access-l78qc\") pod \"limitador-operator-controller-manager-856dd584c5-q9l85\" (UID: \"6754baae-fae7-40dd-ab9b-54fd68603755\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-q9l85"
Oct 29 22:21:54.676313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.676241    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l78qc\" (UniqueName: \"kubernetes.io/projected/6754baae-fae7-40dd-ab9b-54fd68603755-kube-api-access-l78qc\") pod \"limitador-operator-controller-manager-856dd584c5-q9l85\" (UID: \"6754baae-fae7-40dd-ab9b-54fd68603755\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-q9l85"
Oct 29 22:21:54.697851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.697824    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:54.700512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.700493    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"
Oct 29 22:21:54.711114 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:21:54.711083    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-qcwsw\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="11ea846f-9175-4f0b-9e75-d6647682bda1" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"
Oct 29 22:21:54.724223 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:21:54.724192    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-t87wd\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="88927588-2efd-4b81-81c5-a7ced3827020" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:54.737417 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:21:54.737381    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-qcwsw\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="11ea846f-9175-4f0b-9e75-d6647682bda1" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"
Oct 29 22:21:54.750404 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:21:54.750370    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-t87wd\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="88927588-2efd-4b81-81c5-a7ced3827020" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:54.756693 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.756665    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88927588-2efd-4b81-81c5-a7ced3827020-kube-api-access-2tdx4\" (UniqueName: \"kubernetes.io/projected/88927588-2efd-4b81-81c5-a7ced3827020-kube-api-access-2tdx4\") pod \"88927588-2efd-4b81-81c5-a7ced3827020\" (UID: \"88927588-2efd-4b81-81c5-a7ced3827020\") "
Oct 29 22:21:54.758978 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.758958    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88927588-2efd-4b81-81c5-a7ced3827020-kube-api-access-2tdx4" pod "88927588-2efd-4b81-81c5-a7ced3827020" (UID: "88927588-2efd-4b81-81c5-a7ced3827020"). InnerVolumeSpecName "kube-api-access-2tdx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:21:54.841528 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.841495    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"
Oct 29 22:21:54.848507 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.848486    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-q9l85"
Oct 29 22:21:54.857722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.857691    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88927588-2efd-4b81-81c5-a7ced3827020-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88927588-2efd-4b81-81c5-a7ced3827020-extensions-socket-volume\") pod \"88927588-2efd-4b81-81c5-a7ced3827020\" (UID: \"88927588-2efd-4b81-81c5-a7ced3827020\") "
Oct 29 22:21:54.857881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.857746    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ea846f-9175-4f0b-9e75-d6647682bda1-kube-api-access-8kwlk\" (UniqueName: \"kubernetes.io/projected/11ea846f-9175-4f0b-9e75-d6647682bda1-kube-api-access-8kwlk\") pod \"11ea846f-9175-4f0b-9e75-d6647682bda1\" (UID: \"11ea846f-9175-4f0b-9e75-d6647682bda1\") "
Oct 29 22:21:54.857982 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.857963    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tdx4\" (UniqueName: \"kubernetes.io/projected/88927588-2efd-4b81-81c5-a7ced3827020-kube-api-access-2tdx4\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:21:54.858333 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.858305    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88927588-2efd-4b81-81c5-a7ced3827020-extensions-socket-volume" pod "88927588-2efd-4b81-81c5-a7ced3827020" (UID: "88927588-2efd-4b81-81c5-a7ced3827020"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:21:54.859995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.859971    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ea846f-9175-4f0b-9e75-d6647682bda1-kube-api-access-8kwlk" pod "11ea846f-9175-4f0b-9e75-d6647682bda1" (UID: "11ea846f-9175-4f0b-9e75-d6647682bda1"). InnerVolumeSpecName "kube-api-access-8kwlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:21:54.958723 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.958640    4559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88927588-2efd-4b81-81c5-a7ced3827020-extensions-socket-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:21:54.958723 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:54.958671    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kwlk\" (UniqueName: \"kubernetes.io/projected/11ea846f-9175-4f0b-9e75-d6647682bda1-kube-api-access-8kwlk\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:21:55.071931 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:55.071899    4559 generic.go:356] "Generic (PLEG): container finished" podID="11ea846f-9175-4f0b-9e75-d6647682bda1" containerID="e98da5dce02aa44e4aef57ce03f58b9428d8709432cf9de4c25ea991b92fa4c7" exitCode=0
Oct 29 22:21:55.072141 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:55.071961    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"
Oct 29 22:21:55.072141 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:55.072004    4559 scope.go:122] "RemoveContainer" containerID="e98da5dce02aa44e4aef57ce03f58b9428d8709432cf9de4c25ea991b92fa4c7"
Oct 29 22:21:55.073601 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:55.073556    4559 generic.go:356] "Generic (PLEG): container finished" podID="88927588-2efd-4b81-81c5-a7ced3827020" containerID="9b121d0ccd0a7a0989819ed1de95efd4a6f3c7bcacc911ef1b34d3379ed72a4b" exitCode=0
Oct 29 22:21:55.073753 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:55.073619    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:55.080987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:55.080912    4559 scope.go:122] "RemoveContainer" containerID="e98da5dce02aa44e4aef57ce03f58b9428d8709432cf9de4c25ea991b92fa4c7"
Oct 29 22:21:55.081316 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:21:55.081292    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e98da5dce02aa44e4aef57ce03f58b9428d8709432cf9de4c25ea991b92fa4c7\": container with ID starting with e98da5dce02aa44e4aef57ce03f58b9428d8709432cf9de4c25ea991b92fa4c7 not found: ID does not exist" containerID="e98da5dce02aa44e4aef57ce03f58b9428d8709432cf9de4c25ea991b92fa4c7"
Oct 29 22:21:55.081408 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:55.081329    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e98da5dce02aa44e4aef57ce03f58b9428d8709432cf9de4c25ea991b92fa4c7"} err="failed to get container status \"e98da5dce02aa44e4aef57ce03f58b9428d8709432cf9de4c25ea991b92fa4c7\": rpc error: code = NotFound desc = could not find container \"e98da5dce02aa44e4aef57ce03f58b9428d8709432cf9de4c25ea991b92fa4c7\": container with ID starting with e98da5dce02aa44e4aef57ce03f58b9428d8709432cf9de4c25ea991b92fa4c7 not found: ID does not exist"
Oct 29 22:21:55.081408 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:55.081347    4559 scope.go:122] "RemoveContainer" containerID="9b121d0ccd0a7a0989819ed1de95efd4a6f3c7bcacc911ef1b34d3379ed72a4b"
Oct 29 22:21:55.085529 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:21:55.085503    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-qcwsw\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="11ea846f-9175-4f0b-9e75-d6647682bda1" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"
Oct 29 22:21:55.089478 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:55.089462    4559 scope.go:122] "RemoveContainer" containerID="9b121d0ccd0a7a0989819ed1de95efd4a6f3c7bcacc911ef1b34d3379ed72a4b"
Oct 29 22:21:55.089734 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:21:55.089712    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b121d0ccd0a7a0989819ed1de95efd4a6f3c7bcacc911ef1b34d3379ed72a4b\": container with ID starting with 9b121d0ccd0a7a0989819ed1de95efd4a6f3c7bcacc911ef1b34d3379ed72a4b not found: ID does not exist" containerID="9b121d0ccd0a7a0989819ed1de95efd4a6f3c7bcacc911ef1b34d3379ed72a4b"
Oct 29 22:21:55.089783 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:55.089741    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b121d0ccd0a7a0989819ed1de95efd4a6f3c7bcacc911ef1b34d3379ed72a4b"} err="failed to get container status \"9b121d0ccd0a7a0989819ed1de95efd4a6f3c7bcacc911ef1b34d3379ed72a4b\": rpc error: code = NotFound desc = could not find container \"9b121d0ccd0a7a0989819ed1de95efd4a6f3c7bcacc911ef1b34d3379ed72a4b\": container with ID starting with 9b121d0ccd0a7a0989819ed1de95efd4a6f3c7bcacc911ef1b34d3379ed72a4b not found: ID does not exist"
Oct 29 22:21:55.098694 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:21:55.098670    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-t87wd\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="88927588-2efd-4b81-81c5-a7ced3827020" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:55.111803 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:21:55.111774    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-qcwsw\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="11ea846f-9175-4f0b-9e75-d6647682bda1" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qcwsw"
Oct 29 22:21:55.124929 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:21:55.124902    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-t87wd\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="88927588-2efd-4b81-81c5-a7ced3827020" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t87wd"
Oct 29 22:21:55.171099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:55.171074    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"]
Oct 29 22:21:55.173160 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:21:55.173128    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3758ef38_0907_46fe_b936_c3c62361f2bb.slice/crio-357bbcd42702c2a3d1fd6ee45098045318534ef0d3f48cff06bed8404b85b929 WatchSource:0}: Error finding container 357bbcd42702c2a3d1fd6ee45098045318534ef0d3f48cff06bed8404b85b929: Status 404 returned error can't find the container with id 357bbcd42702c2a3d1fd6ee45098045318534ef0d3f48cff06bed8404b85b929
Oct 29 22:21:55.174700 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:55.174677    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-q9l85"]
Oct 29 22:21:55.186147 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:21:55.186119    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6754baae_fae7_40dd_ab9b_54fd68603755.slice/crio-64198d255a2ed145c2589aa8f302cbb326f153d43483e513fd2e17565c6f50ef WatchSource:0}: Error finding container 64198d255a2ed145c2589aa8f302cbb326f153d43483e513fd2e17565c6f50ef: Status 404 returned error can't find the container with id 64198d255a2ed145c2589aa8f302cbb326f153d43483e513fd2e17565c6f50ef
Oct 29 22:21:55.225861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:55.225836    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ea846f-9175-4f0b-9e75-d6647682bda1" path="/var/lib/kubelet/pods/11ea846f-9175-4f0b-9e75-d6647682bda1/volumes"
Oct 29 22:21:55.226223 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:55.226177    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88927588-2efd-4b81-81c5-a7ced3827020" path="/var/lib/kubelet/pods/88927588-2efd-4b81-81c5-a7ced3827020/volumes"
Oct 29 22:21:56.081928 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:56.081886    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-q9l85" event={"ID":"6754baae-fae7-40dd-ab9b-54fd68603755","Type":"ContainerStarted","Data":"f9c4b0c4446a9194254f96532e7b6027caba42562c8fd7a4cef157ec9380a966"}
Oct 29 22:21:56.082563 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:56.081957    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-q9l85" event={"ID":"6754baae-fae7-40dd-ab9b-54fd68603755","Type":"ContainerStarted","Data":"64198d255a2ed145c2589aa8f302cbb326f153d43483e513fd2e17565c6f50ef"}
Oct 29 22:21:56.082563 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:56.082082    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-q9l85"
Oct 29 22:21:56.083563 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:56.083533    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz" event={"ID":"3758ef38-0907-46fe-b936-c3c62361f2bb","Type":"ContainerStarted","Data":"fbf805f553b2faf7dd35a3226ebe92ea0413c542cf1c4ea2d17df9062ff6bbf7"}
Oct 29 22:21:56.083690 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:56.083581    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz" event={"ID":"3758ef38-0907-46fe-b936-c3c62361f2bb","Type":"ContainerStarted","Data":"357bbcd42702c2a3d1fd6ee45098045318534ef0d3f48cff06bed8404b85b929"}
Oct 29 22:21:56.083690 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:56.083670    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"
Oct 29 22:21:56.124237 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:56.124180    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-q9l85" podStartSLOduration=2.12416566 podStartE2EDuration="2.12416566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:21:56.123267974 +0000 UTC m=+455.034711553" watchObservedRunningTime="2026-10-29 22:21:56.12416566 +0000 UTC m=+455.035609213"
Oct 29 22:21:56.167550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:21:56.167481    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz" podStartSLOduration=2.167464465 podStartE2EDuration="2.167464465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:21:56.165236071 +0000 UTC m=+455.076679626" watchObservedRunningTime="2026-10-29 22:21:56.167464465 +0000 UTC m=+455.078908020"
Oct 29 22:22:07.090650 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:07.090616    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-q9l85"
Oct 29 22:22:07.091199 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:07.090700    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"
Oct 29 22:22:09.267416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.267379    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"]
Oct 29 22:22:09.267889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.267645    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz" podUID="3758ef38-0907-46fe-b936-c3c62361f2bb" containerName="manager" containerID="cri-o://fbf805f553b2faf7dd35a3226ebe92ea0413c542cf1c4ea2d17df9062ff6bbf7" gracePeriod=10
Oct 29 22:22:09.501957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.501927    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"]
Oct 29 22:22:09.523477 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.523409    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"
Oct 29 22:22:09.534289 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.534259    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"]
Oct 29 22:22:09.554782 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.554751    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95b92349-1ece-43d1-aa95-f4ca375813d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-wdqgz\" (UID: \"95b92349-1ece-43d1-aa95-f4ca375813d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"
Oct 29 22:22:09.554933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.554793    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxrmw\" (UniqueName: \"kubernetes.io/projected/95b92349-1ece-43d1-aa95-f4ca375813d0-kube-api-access-jxrmw\") pod \"kuadrant-operator-controller-manager-57bb9bfff-wdqgz\" (UID: \"95b92349-1ece-43d1-aa95-f4ca375813d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"
Oct 29 22:22:09.581675 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.581642    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"
Oct 29 22:22:09.655943 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.655910    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95b92349-1ece-43d1-aa95-f4ca375813d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-wdqgz\" (UID: \"95b92349-1ece-43d1-aa95-f4ca375813d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"
Oct 29 22:22:09.656117 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.655954    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxrmw\" (UniqueName: \"kubernetes.io/projected/95b92349-1ece-43d1-aa95-f4ca375813d0-kube-api-access-jxrmw\") pod \"kuadrant-operator-controller-manager-57bb9bfff-wdqgz\" (UID: \"95b92349-1ece-43d1-aa95-f4ca375813d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"
Oct 29 22:22:09.656298 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.656278    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95b92349-1ece-43d1-aa95-f4ca375813d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-wdqgz\" (UID: \"95b92349-1ece-43d1-aa95-f4ca375813d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"
Oct 29 22:22:09.675884 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.675859    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxrmw\" (UniqueName: \"kubernetes.io/projected/95b92349-1ece-43d1-aa95-f4ca375813d0-kube-api-access-jxrmw\") pod \"kuadrant-operator-controller-manager-57bb9bfff-wdqgz\" (UID: \"95b92349-1ece-43d1-aa95-f4ca375813d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"
Oct 29 22:22:09.756935 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.756892    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3758ef38-0907-46fe-b936-c3c62361f2bb-kube-api-access-7j6cv\" (UniqueName: \"kubernetes.io/projected/3758ef38-0907-46fe-b936-c3c62361f2bb-kube-api-access-7j6cv\") pod \"3758ef38-0907-46fe-b936-c3c62361f2bb\" (UID: \"3758ef38-0907-46fe-b936-c3c62361f2bb\") "
Oct 29 22:22:09.757107 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.756966    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3758ef38-0907-46fe-b936-c3c62361f2bb-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3758ef38-0907-46fe-b936-c3c62361f2bb-extensions-socket-volume\") pod \"3758ef38-0907-46fe-b936-c3c62361f2bb\" (UID: \"3758ef38-0907-46fe-b936-c3c62361f2bb\") "
Oct 29 22:22:09.757362 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.757336    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3758ef38-0907-46fe-b936-c3c62361f2bb-extensions-socket-volume" pod "3758ef38-0907-46fe-b936-c3c62361f2bb" (UID: "3758ef38-0907-46fe-b936-c3c62361f2bb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:22:09.759156 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.759134    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3758ef38-0907-46fe-b936-c3c62361f2bb-kube-api-access-7j6cv" pod "3758ef38-0907-46fe-b936-c3c62361f2bb" (UID: "3758ef38-0907-46fe-b936-c3c62361f2bb"). InnerVolumeSpecName "kube-api-access-7j6cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:22:09.857644 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.857553    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j6cv\" (UniqueName: \"kubernetes.io/projected/3758ef38-0907-46fe-b936-c3c62361f2bb-kube-api-access-7j6cv\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:22:09.857644 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.857589    4559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3758ef38-0907-46fe-b936-c3c62361f2bb-extensions-socket-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:22:09.866494 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:09.866453    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"
Oct 29 22:22:10.147530 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:10.147490    4559 generic.go:356] "Generic (PLEG): container finished" podID="3758ef38-0907-46fe-b936-c3c62361f2bb" containerID="fbf805f553b2faf7dd35a3226ebe92ea0413c542cf1c4ea2d17df9062ff6bbf7" exitCode=0
Oct 29 22:22:10.147719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:10.147572    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz" event={"ID":"3758ef38-0907-46fe-b936-c3c62361f2bb","Type":"ContainerDied","Data":"fbf805f553b2faf7dd35a3226ebe92ea0413c542cf1c4ea2d17df9062ff6bbf7"}
Oct 29 22:22:10.147719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:10.147600    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"
Oct 29 22:22:10.147719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:10.147614    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz" event={"ID":"3758ef38-0907-46fe-b936-c3c62361f2bb","Type":"ContainerDied","Data":"357bbcd42702c2a3d1fd6ee45098045318534ef0d3f48cff06bed8404b85b929"}
Oct 29 22:22:10.147719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:10.147636    4559 scope.go:122] "RemoveContainer" containerID="fbf805f553b2faf7dd35a3226ebe92ea0413c542cf1c4ea2d17df9062ff6bbf7"
Oct 29 22:22:10.156508 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:10.156478    4559 scope.go:122] "RemoveContainer" containerID="fbf805f553b2faf7dd35a3226ebe92ea0413c542cf1c4ea2d17df9062ff6bbf7"
Oct 29 22:22:10.156787 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:22:10.156768    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbf805f553b2faf7dd35a3226ebe92ea0413c542cf1c4ea2d17df9062ff6bbf7\": container with ID starting with fbf805f553b2faf7dd35a3226ebe92ea0413c542cf1c4ea2d17df9062ff6bbf7 not found: ID does not exist" containerID="fbf805f553b2faf7dd35a3226ebe92ea0413c542cf1c4ea2d17df9062ff6bbf7"
Oct 29 22:22:10.156861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:10.156798    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbf805f553b2faf7dd35a3226ebe92ea0413c542cf1c4ea2d17df9062ff6bbf7"} err="failed to get container status \"fbf805f553b2faf7dd35a3226ebe92ea0413c542cf1c4ea2d17df9062ff6bbf7\": rpc error: code = NotFound desc = could not find container \"fbf805f553b2faf7dd35a3226ebe92ea0413c542cf1c4ea2d17df9062ff6bbf7\": container with ID starting with fbf805f553b2faf7dd35a3226ebe92ea0413c542cf1c4ea2d17df9062ff6bbf7 not found: ID does not exist"
Oct 29 22:22:10.196861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:10.196829    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"]
Oct 29 22:22:10.209499 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:10.209462    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-f92lz"]
Oct 29 22:22:10.210973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:10.210950    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"]
Oct 29 22:22:10.214098 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:22:10.214070    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b92349_1ece_43d1_aa95_f4ca375813d0.slice/crio-3b8375058f9c96cab7381d1b302cb29afdd655f123711ba8ad328accae005d5c WatchSource:0}: Error finding container 3b8375058f9c96cab7381d1b302cb29afdd655f123711ba8ad328accae005d5c: Status 404 returned error can't find the container with id 3b8375058f9c96cab7381d1b302cb29afdd655f123711ba8ad328accae005d5c
Oct 29 22:22:11.153776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:11.153740    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz" event={"ID":"95b92349-1ece-43d1-aa95-f4ca375813d0","Type":"ContainerStarted","Data":"0deeff0b1883775a1f5887b769cbde31582d1501679f3196216ea9fc7bcd1b5e"}
Oct 29 22:22:11.154294 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:11.153803    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz" event={"ID":"95b92349-1ece-43d1-aa95-f4ca375813d0","Type":"ContainerStarted","Data":"3b8375058f9c96cab7381d1b302cb29afdd655f123711ba8ad328accae005d5c"}
Oct 29 22:22:11.154294 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:11.153941    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"
Oct 29 22:22:11.217308 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:11.217245    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz" podStartSLOduration=2.217230029 podStartE2EDuration="2.217230029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:11.204832158 +0000 UTC m=+470.116275703" watchObservedRunningTime="2026-10-29 22:22:11.217230029 +0000 UTC m=+470.128673582"
Oct 29 22:22:11.226446 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:11.226411    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3758ef38-0907-46fe-b936-c3c62361f2bb" path="/var/lib/kubelet/pods/3758ef38-0907-46fe-b936-c3c62361f2bb/volumes"
Oct 29 22:22:13.692314 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:22:13.692276    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:22:22.166633 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:22.166592    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"
Oct 29 22:22:41.516371 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:41.516258    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:22:41.516785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:41.516648    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3758ef38-0907-46fe-b936-c3c62361f2bb" containerName="manager"
Oct 29 22:22:41.516785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:41.516662    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3758ef38-0907-46fe-b936-c3c62361f2bb" containerName="manager"
Oct 29 22:22:41.516785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:41.516732    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3758ef38-0907-46fe-b936-c3c62361f2bb" containerName="manager"
Oct 29 22:22:41.519828 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:41.519793    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-s75k9"
Oct 29 22:22:41.534176 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:41.534149    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 29 22:22:41.546147 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:41.546123    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qwmm7\""
Oct 29 22:22:41.547662 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:41.547642    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:22:41.590626 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:41.590590    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:22:41.625617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:41.625583    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnkwj\" (UniqueName: \"kubernetes.io/projected/81712066-dc50-44ed-a65f-820134d46bbd-kube-api-access-fnkwj\") pod \"limitador-limitador-7c9b87d574-s75k9\" (UID: \"81712066-dc50-44ed-a65f-820134d46bbd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-s75k9"
Oct 29 22:22:41.625617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:41.625618    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81712066-dc50-44ed-a65f-820134d46bbd-config-file\") pod \"limitador-limitador-7c9b87d574-s75k9\" (UID: \"81712066-dc50-44ed-a65f-820134d46bbd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-s75k9"
Oct 29 22:22:41.726898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:41.726850    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnkwj\" (UniqueName: \"kubernetes.io/projected/81712066-dc50-44ed-a65f-820134d46bbd-kube-api-access-fnkwj\") pod \"limitador-limitador-7c9b87d574-s75k9\" (UID: \"81712066-dc50-44ed-a65f-820134d46bbd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-s75k9"
Oct 29 22:22:41.726898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:41.726902    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81712066-dc50-44ed-a65f-820134d46bbd-config-file\") pod \"limitador-limitador-7c9b87d574-s75k9\" (UID: \"81712066-dc50-44ed-a65f-820134d46bbd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-s75k9"
Oct 29 22:22:41.727546 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:41.727517    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81712066-dc50-44ed-a65f-820134d46bbd-config-file\") pod \"limitador-limitador-7c9b87d574-s75k9\" (UID: \"81712066-dc50-44ed-a65f-820134d46bbd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-s75k9"
Oct 29 22:22:41.746540 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:41.746511    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnkwj\" (UniqueName: \"kubernetes.io/projected/81712066-dc50-44ed-a65f-820134d46bbd-kube-api-access-fnkwj\") pod \"limitador-limitador-7c9b87d574-s75k9\" (UID: \"81712066-dc50-44ed-a65f-820134d46bbd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-s75k9"
Oct 29 22:22:41.832714 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:41.832611    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-s75k9"
Oct 29 22:22:42.130663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:42.130638    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:22:42.132842 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:22:42.132793    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81712066_dc50_44ed_a65f_820134d46bbd.slice/crio-6e5b72e371731953d65201d84946d949c3e4a14fa977e2f612c0778866084f91 WatchSource:0}: Error finding container 6e5b72e371731953d65201d84946d949c3e4a14fa977e2f612c0778866084f91: Status 404 returned error can't find the container with id 6e5b72e371731953d65201d84946d949c3e4a14fa977e2f612c0778866084f91
Oct 29 22:22:42.292419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:42.292382    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-s75k9" event={"ID":"81712066-dc50-44ed-a65f-820134d46bbd","Type":"ContainerStarted","Data":"6e5b72e371731953d65201d84946d949c3e4a14fa977e2f612c0778866084f91"}
Oct 29 22:22:43.299318 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:43.299283    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-s75k9" event={"ID":"81712066-dc50-44ed-a65f-820134d46bbd","Type":"ContainerStarted","Data":"ae0e19d9abd297ee0c2f3d8de7e18ce0337a54f7d2fe836c84054363737125cd"}
Oct 29 22:22:43.299687 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:43.299402    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-s75k9"
Oct 29 22:22:43.336559 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:43.336506    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-s75k9" podStartSLOduration=1.357184063 podStartE2EDuration="2.336491408s" totalImagesPullingTime="979.307345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:43.335686341 +0000 UTC m=+502.247129895" watchObservedRunningTime="2026-10-29 22:22:43.336491408 +0000 UTC m=+502.247934962"
Oct 29 22:22:43.693292 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:22:43.693255    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:22:54.304916 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:22:54.304882    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-s75k9"
Oct 29 22:23:13.692648 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:23:13.692603    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:23:43.692090 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:23:43.691990    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:24:04.089103 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:24:04.089010    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:24:13.693119 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:24:13.693084    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:24:21.275025 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:24:21.274994    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:24:21.276162 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:24:21.276139    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:24:43.692525 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:24:43.692473    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:25:06.103556 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:06.103512    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:25:09.943772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:09.943736    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:25:13.692190 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:25:13.692156    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:25:13.966766 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:13.966688    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"]
Oct 29 22:25:13.970429 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:13.970410    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.003095 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.003062    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-grkds\""
Oct 29 22:25:14.003293 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.003142    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 29 22:25:14.004611 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.004581    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"]
Oct 29 22:25:14.044971 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.044940    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 29 22:25:14.045208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.044994    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 29 22:25:14.066417 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.066374    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.066417 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.066412    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.066663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.066447    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.066663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.066524    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.066663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.066547    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44p9r\" (UniqueName: \"kubernetes.io/projected/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-kube-api-access-44p9r\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.066663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.066599    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.167259 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.167219    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.167259 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.167257    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.167546 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.167277    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.167546 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.167304    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.167546 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.167328    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44p9r\" (UniqueName: \"kubernetes.io/projected/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-kube-api-access-44p9r\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.167546 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.167376    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.167770 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.167733    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.167836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.167785    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.167890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.167860    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.169732 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.169708    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.170031 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.170016    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.184234 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.184210    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:25:14.188556 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.188528    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44p9r\" (UniqueName: \"kubernetes.io/projected/b2faf2e5-b0dd-4eb3-a396-ed3de57a737b-kube-api-access-44p9r\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rm75q\" (UID: \"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.284581 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.284489    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:14.587061 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.587033    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"]
Oct 29 22:25:14.589200 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:25:14.589172    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2faf2e5_b0dd_4eb3_a396_ed3de57a737b.slice/crio-c0b546650a2c767a1523ab49ffe732217c0e09271f3a9db6aa64202709ecaab2 WatchSource:0}: Error finding container c0b546650a2c767a1523ab49ffe732217c0e09271f3a9db6aa64202709ecaab2: Status 404 returned error can't find the container with id c0b546650a2c767a1523ab49ffe732217c0e09271f3a9db6aa64202709ecaab2
Oct 29 22:25:14.953047 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:14.953006    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q" event={"ID":"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b","Type":"ContainerStarted","Data":"c0b546650a2c767a1523ab49ffe732217c0e09271f3a9db6aa64202709ecaab2"}
Oct 29 22:25:15.959251 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:15.959151    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q" event={"ID":"b2faf2e5-b0dd-4eb3-a396-ed3de57a737b","Type":"ContainerStarted","Data":"15e370a4053c84ca7e51d904cd50c4d8c21bf3b5151984ad153064949beb95ff"}
Oct 29 22:25:16.005306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.005252    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q" podStartSLOduration=1.8902199240000002 podStartE2EDuration="3.005237544s" totalImagesPullingTime="1.11501762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:16.003924536 +0000 UTC m=+654.915368092" watchObservedRunningTime="2026-10-29 22:25:16.005237544 +0000 UTC m=+654.916681098"
Oct 29 22:25:16.276988 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.276905    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"]
Oct 29 22:25:16.280520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.280499    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.285537 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.285513    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:16.302561 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.302524    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"]
Oct 29 22:25:16.313739 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.313702    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 29 22:25:16.387753 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.387715    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e62d21f0-40a0-471a-9148-d5374553ee04-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.387753 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.387758    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnszj\" (UniqueName: \"kubernetes.io/projected/e62d21f0-40a0-471a-9148-d5374553ee04-kube-api-access-pnszj\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.388051 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.387786    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e62d21f0-40a0-471a-9148-d5374553ee04-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.388051 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.387876    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62d21f0-40a0-471a-9148-d5374553ee04-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.388051 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.387911    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62d21f0-40a0-471a-9148-d5374553ee04-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.388051 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.388008    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62d21f0-40a0-471a-9148-d5374553ee04-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.488974 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.488926    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e62d21f0-40a0-471a-9148-d5374553ee04-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.488974 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.488972    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62d21f0-40a0-471a-9148-d5374553ee04-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.489229 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.488999    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62d21f0-40a0-471a-9148-d5374553ee04-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.489229 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.489041    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62d21f0-40a0-471a-9148-d5374553ee04-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.489229 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.489134    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e62d21f0-40a0-471a-9148-d5374553ee04-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.489229 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.489160    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnszj\" (UniqueName: \"kubernetes.io/projected/e62d21f0-40a0-471a-9148-d5374553ee04-kube-api-access-pnszj\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.489420 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.489395    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e62d21f0-40a0-471a-9148-d5374553ee04-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.489475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.489424    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62d21f0-40a0-471a-9148-d5374553ee04-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.489475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.489455    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62d21f0-40a0-471a-9148-d5374553ee04-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.491516 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.491489    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62d21f0-40a0-471a-9148-d5374553ee04-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.492022 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.492005    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e62d21f0-40a0-471a-9148-d5374553ee04-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.510199 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.510164    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnszj\" (UniqueName: \"kubernetes.io/projected/e62d21f0-40a0-471a-9148-d5374553ee04-kube-api-access-pnszj\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5r7kn\" (UID: \"e62d21f0-40a0-471a-9148-d5374553ee04\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.593390 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.593304    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:16.893335 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.893312    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"]
Oct 29 22:25:16.895431 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:25:16.895406    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62d21f0_40a0_471a_9148_d5374553ee04.slice/crio-b7335b2e5141138707eeab2aacc0c1ab660aea7c89c02c565ad84e65d631144e WatchSource:0}: Error finding container b7335b2e5141138707eeab2aacc0c1ab660aea7c89c02c565ad84e65d631144e: Status 404 returned error can't find the container with id b7335b2e5141138707eeab2aacc0c1ab660aea7c89c02c565ad84e65d631144e
Oct 29 22:25:16.964648 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:16.964616    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn" event={"ID":"e62d21f0-40a0-471a-9148-d5374553ee04","Type":"ContainerStarted","Data":"b7335b2e5141138707eeab2aacc0c1ab660aea7c89c02c565ad84e65d631144e"}
Oct 29 22:25:17.658828 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:17.658787    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:25:17.970445 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:17.970342    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn" event={"ID":"e62d21f0-40a0-471a-9148-d5374553ee04","Type":"ContainerStarted","Data":"2f8ac12bb9dd83235707cb67b4e097d12b058354279ebc3ae7ced521b468f77b"}
Oct 29 22:25:18.014434 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:18.014381    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn" podStartSLOduration=1.7878183619999999 podStartE2EDuration="2.014367682s" totalImagesPullingTime="226.54932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:18.013073506 +0000 UTC m=+656.924517060" watchObservedRunningTime="2026-10-29 22:25:18.014367682 +0000 UTC m=+656.925811236"
Oct 29 22:25:18.594221 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:18.594187    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:19.306653 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.306620    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"]
Oct 29 22:25:19.310980 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.310956    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.315428 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.315401    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj6zm\" (UniqueName: \"kubernetes.io/projected/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-kube-api-access-kj6zm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.315560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.315434    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.315560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.315523    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.315663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.315575    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.315724 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.315700    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.315842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.315796    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.331876 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.331845    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"]
Oct 29 22:25:19.341776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.341752    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 29 22:25:19.416404 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.416371    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.416569 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.416412    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.416569 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.416435    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.416569 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.416462    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.416569 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.416513    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj6zm\" (UniqueName: \"kubernetes.io/projected/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-kube-api-access-kj6zm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.416569 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.416547    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.416900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.416876    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.417000 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.416905    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.417062 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.416992    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.418972 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.418953    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.419345 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.419325    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.437132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.437096    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj6zm\" (UniqueName: \"kubernetes.io/projected/91680d4e-1b82-4456-96cb-0fa36ae5ff4f-kube-api-access-kj6zm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t\" (UID: \"91680d4e-1b82-4456-96cb-0fa36ae5ff4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.625299 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.625209    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:19.924786 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.924758    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"]
Oct 29 22:25:19.926423 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:25:19.926394    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91680d4e_1b82_4456_96cb_0fa36ae5ff4f.slice/crio-a96a9395b647de212d1f5e5a7ea458420205ed076d73fc2953dbf06ed59bb9c9 WatchSource:0}: Error finding container a96a9395b647de212d1f5e5a7ea458420205ed076d73fc2953dbf06ed59bb9c9: Status 404 returned error can't find the container with id a96a9395b647de212d1f5e5a7ea458420205ed076d73fc2953dbf06ed59bb9c9
Oct 29 22:25:19.979248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:19.979210    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t" event={"ID":"91680d4e-1b82-4456-96cb-0fa36ae5ff4f","Type":"ContainerStarted","Data":"a96a9395b647de212d1f5e5a7ea458420205ed076d73fc2953dbf06ed59bb9c9"}
Oct 29 22:25:20.985660 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:20.985617    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t" event={"ID":"91680d4e-1b82-4456-96cb-0fa36ae5ff4f","Type":"ContainerStarted","Data":"95fc819e3ee043071fadd36747533a7584d05c0642459ae7eb04fe502d79f3eb"}
Oct 29 22:25:21.028380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:21.028325    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t" podStartSLOduration=1.79962912 podStartE2EDuration="2.02830844s" totalImagesPullingTime="228.67932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:21.02696296 +0000 UTC m=+659.938406513" watchObservedRunningTime="2026-10-29 22:25:21.02830844 +0000 UTC m=+659.939751994"
Oct 29 22:25:21.626138 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:21.626095    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:24.285237 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:24.285197    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:24.297986 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:24.297955    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:24.403186 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:24.403154    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:25:25.015880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:25.015850    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rm75q"
Oct 29 22:25:26.593800 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.593709    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:26.611656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.611630    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:26.810873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.810844    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"]
Oct 29 22:25:26.814825 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.814788    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.867763 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.867676    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"]
Oct 29 22:25:26.883541 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.883504    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.883708 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.883548    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pszvl\" (UniqueName: \"kubernetes.io/projected/b15b2407-227c-40f8-95a9-4bfad483a2bb-kube-api-access-pszvl\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.883708 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.883581    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.883791 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.883699    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.883844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.883782    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.883891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.883864    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.883971 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.883947    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15b2407-227c-40f8-95a9-4bfad483a2bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.884028 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.884009    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.884092 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.884072    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.916499 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.916469    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-67ntj\""
Oct 29 22:25:26.984742 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.984689    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.984742 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.984746    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pszvl\" (UniqueName: \"kubernetes.io/projected/b15b2407-227c-40f8-95a9-4bfad483a2bb-kube-api-access-pszvl\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.985020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.984790    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.985020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.984826    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.985020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.984873    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.985020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.984898    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.985020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.984951    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15b2407-227c-40f8-95a9-4bfad483a2bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.985020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.985002    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.985306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.985027    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.985306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.985257    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.985612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.985556    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.985612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.985585    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.985952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.985688    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15b2407-227c-40f8-95a9-4bfad483a2bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.985952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.985778    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.987436 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.987413    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:26.987620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:26.987603    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:27.006591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:27.006554    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:27.006894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:27.006871    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pszvl\" (UniqueName: \"kubernetes.io/projected/b15b2407-227c-40f8-95a9-4bfad483a2bb-kube-api-access-pszvl\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-k9fn7\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:27.028715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:27.028686    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5r7kn"
Oct 29 22:25:27.128206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:27.128110    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:27.439147 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:27.439067    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"]
Oct 29 22:25:27.441480 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:25:27.441456    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15b2407_227c_40f8_95a9_4bfad483a2bb.slice/crio-7daed353eda6682315cf42a70d04f76cc4014182a06a0a9c9eed297f9cc3ae8c WatchSource:0}: Error finding container 7daed353eda6682315cf42a70d04f76cc4014182a06a0a9c9eed297f9cc3ae8c: Status 404 returned error can't find the container with id 7daed353eda6682315cf42a70d04f76cc4014182a06a0a9c9eed297f9cc3ae8c
Oct 29 22:25:28.023215 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:28.023181    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7" event={"ID":"b15b2407-227c-40f8-95a9-4bfad483a2bb","Type":"ContainerStarted","Data":"7daed353eda6682315cf42a70d04f76cc4014182a06a0a9c9eed297f9cc3ae8c"}
Oct 29 22:25:29.626165 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:29.626132    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:29.640492 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:29.640458    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:30.035544 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:30.035508    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:25:30.035686 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:30.035594    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:25:30.035686 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:30.035636    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:25:30.046856 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:30.046829    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t"
Oct 29 22:25:31.062290 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:31.062173    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7" event={"ID":"b15b2407-227c-40f8-95a9-4bfad483a2bb","Type":"ContainerStarted","Data":"2b0b55419a6269f1996fd04025fef6fff0f2f58852f95c980771fa74ea20f055"}
Oct 29 22:25:31.122068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:31.122017    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7" podStartSLOduration=2.530783162 podStartE2EDuration="5.122002562s" totalImagesPullingTime="2.5912194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:31.120101352 +0000 UTC m=+670.031544906" watchObservedRunningTime="2026-10-29 22:25:31.122002562 +0000 UTC m=+670.033446115"
Oct 29 22:25:31.128652 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:31.128624    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:31.129794 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:31.129765    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:25:31.129920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:31.129837    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7" podUID="b15b2407-227c-40f8-95a9-4bfad483a2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:25:32.129115 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:32.129082    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:25:32.129492 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:32.129138    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7" podUID="b15b2407-227c-40f8-95a9-4bfad483a2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:25:33.128702 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:33.128623    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:25:33.128702 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:33.128688    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7" podUID="b15b2407-227c-40f8-95a9-4bfad483a2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:25:34.129313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:34.129271    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:25:34.129716 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:34.129323    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7" podUID="b15b2407-227c-40f8-95a9-4bfad483a2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:25:35.128599 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:35.128563    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:25:35.128772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:35.128620    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7" podUID="b15b2407-227c-40f8-95a9-4bfad483a2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:25:36.128998 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:36.128952    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:25:36.129445 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:36.129024    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7" podUID="b15b2407-227c-40f8-95a9-4bfad483a2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:25:37.128708 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:37.128669    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:25:37.128995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:37.128724    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7" podUID="b15b2407-227c-40f8-95a9-4bfad483a2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:25:37.849100 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:37.849058    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"]
Oct 29 22:25:37.853383 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:37.853361    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"
Oct 29 22:25:37.873842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:37.873725    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"]
Oct 29 22:25:37.885105 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:37.885070    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-2f5bf\""
Oct 29 22:25:37.887102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:37.887081    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 29 22:25:37.887258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:37.887106    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 29 22:25:38.015402 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:38.015353    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b4e73d-dc80-4c28-8c48-d1248bc687b4-metrics-certs\") pod \"ai-gateway-controller-55b4c89cbd-6szvd\" (UID: \"d5b4e73d-dc80-4c28-8c48-d1248bc687b4\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"
Oct 29 22:25:38.015402 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:38.015400    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2gtn\" (UniqueName: \"kubernetes.io/projected/d5b4e73d-dc80-4c28-8c48-d1248bc687b4-kube-api-access-t2gtn\") pod \"ai-gateway-controller-55b4c89cbd-6szvd\" (UID: \"d5b4e73d-dc80-4c28-8c48-d1248bc687b4\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"
Oct 29 22:25:38.116663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:38.116562    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b4e73d-dc80-4c28-8c48-d1248bc687b4-metrics-certs\") pod \"ai-gateway-controller-55b4c89cbd-6szvd\" (UID: \"d5b4e73d-dc80-4c28-8c48-d1248bc687b4\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"
Oct 29 22:25:38.116663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:38.116596    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2gtn\" (UniqueName: \"kubernetes.io/projected/d5b4e73d-dc80-4c28-8c48-d1248bc687b4-kube-api-access-t2gtn\") pod \"ai-gateway-controller-55b4c89cbd-6szvd\" (UID: \"d5b4e73d-dc80-4c28-8c48-d1248bc687b4\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"
Oct 29 22:25:38.119261 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:38.119234    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b4e73d-dc80-4c28-8c48-d1248bc687b4-metrics-certs\") pod \"ai-gateway-controller-55b4c89cbd-6szvd\" (UID: \"d5b4e73d-dc80-4c28-8c48-d1248bc687b4\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"
Oct 29 22:25:38.137591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:38.137554    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2gtn\" (UniqueName: \"kubernetes.io/projected/d5b4e73d-dc80-4c28-8c48-d1248bc687b4-kube-api-access-t2gtn\") pod \"ai-gateway-controller-55b4c89cbd-6szvd\" (UID: \"d5b4e73d-dc80-4c28-8c48-d1248bc687b4\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"
Oct 29 22:25:38.167705 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:38.167666    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"
Oct 29 22:25:38.425250 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:38.424493    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:38.425250 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:38.425106    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:38.426747 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:38.426724    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:25:38.473863 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:38.473837    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"]
Oct 29 22:25:38.476090 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:25:38.476066    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b4e73d_dc80_4c28_8c48_d1248bc687b4.slice/crio-4f93c3dc0a9a679bd41c6a34f1cbad93b2ccd62270bef0a8b9464ba6a1da8800 WatchSource:0}: Error finding container 4f93c3dc0a9a679bd41c6a34f1cbad93b2ccd62270bef0a8b9464ba6a1da8800: Status 404 returned error can't find the container with id 4f93c3dc0a9a679bd41c6a34f1cbad93b2ccd62270bef0a8b9464ba6a1da8800
Oct 29 22:25:39.104701 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:39.104666    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd" event={"ID":"d5b4e73d-dc80-4c28-8c48-d1248bc687b4","Type":"ContainerStarted","Data":"4f93c3dc0a9a679bd41c6a34f1cbad93b2ccd62270bef0a8b9464ba6a1da8800"}
Oct 29 22:25:41.122537 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:41.122501    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd" event={"ID":"d5b4e73d-dc80-4c28-8c48-d1248bc687b4","Type":"ContainerStarted","Data":"d4a5cc6889a0d0b661cc0a5df88268d05b20155e4b75c3f87729749061ce0552"}
Oct 29 22:25:41.122991 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:41.122621    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"
Oct 29 22:25:41.167084 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:41.167029    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd" podStartSLOduration=1.702365567 podStartE2EDuration="4.167015395s" totalImagesPullingTime="2.464649828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:41.165897961 +0000 UTC m=+680.077341515" watchObservedRunningTime="2026-10-29 22:25:41.167015395 +0000 UTC m=+680.078458949"
Oct 29 22:25:41.807430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:41.807399    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"]
Oct 29 22:25:41.811118 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:41.811093    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:41.843393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:41.843354    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"]
Oct 29 22:25:41.953612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:41.953574    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh2js\" (UniqueName: \"kubernetes.io/projected/d166a89c-98b6-4209-b8d8-5b405090031b-kube-api-access-sh2js\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:41.953775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:41.953691    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d166a89c-98b6-4209-b8d8-5b405090031b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:41.953775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:41.953753    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d166a89c-98b6-4209-b8d8-5b405090031b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:41.953924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:41.953784    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d166a89c-98b6-4209-b8d8-5b405090031b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:41.953965 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:41.953932    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d166a89c-98b6-4209-b8d8-5b405090031b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:41.953965 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:41.953962    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d166a89c-98b6-4209-b8d8-5b405090031b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:41.954044 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:41.953984    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d166a89c-98b6-4209-b8d8-5b405090031b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:41.954080 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:41.954062    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d166a89c-98b6-4209-b8d8-5b405090031b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:41.954114 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:41.954096    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d166a89c-98b6-4209-b8d8-5b405090031b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.055509 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.055475    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d166a89c-98b6-4209-b8d8-5b405090031b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.055509 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.055512    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d166a89c-98b6-4209-b8d8-5b405090031b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.055857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.055537    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d166a89c-98b6-4209-b8d8-5b405090031b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.055857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.055583    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d166a89c-98b6-4209-b8d8-5b405090031b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.055857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.055611    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d166a89c-98b6-4209-b8d8-5b405090031b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.055857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.055672    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d166a89c-98b6-4209-b8d8-5b405090031b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.056085 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.055914    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d166a89c-98b6-4209-b8d8-5b405090031b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.056085 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.055983    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d166a89c-98b6-4209-b8d8-5b405090031b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.056085 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.056043    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d166a89c-98b6-4209-b8d8-5b405090031b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.056236 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.056086    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d166a89c-98b6-4209-b8d8-5b405090031b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.056236 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.056164    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh2js\" (UniqueName: \"kubernetes.io/projected/d166a89c-98b6-4209-b8d8-5b405090031b-kube-api-access-sh2js\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.056362 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.056339    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d166a89c-98b6-4209-b8d8-5b405090031b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.056508 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.056487    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d166a89c-98b6-4209-b8d8-5b405090031b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.056666 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.056649    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d166a89c-98b6-4209-b8d8-5b405090031b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.058884 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.058802    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d166a89c-98b6-4209-b8d8-5b405090031b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.059191 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.059167    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d166a89c-98b6-4209-b8d8-5b405090031b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.077674 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.077636    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d166a89c-98b6-4209-b8d8-5b405090031b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.077876 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.077855    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh2js\" (UniqueName: \"kubernetes.io/projected/d166a89c-98b6-4209-b8d8-5b405090031b-kube-api-access-sh2js\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-m55mt\" (UID: \"d166a89c-98b6-4209-b8d8-5b405090031b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.124605 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.124568    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:42.445551 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.445514    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"]
Oct 29 22:25:42.449689 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:25:42.449661    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd166a89c_98b6_4209_b8d8_5b405090031b.slice/crio-9e5bc8bba2f7f79b6026384d6c8cb2a445852c653774cf9f6af3038c6199730e WatchSource:0}: Error finding container 9e5bc8bba2f7f79b6026384d6c8cb2a445852c653774cf9f6af3038c6199730e: Status 404 returned error can't find the container with id 9e5bc8bba2f7f79b6026384d6c8cb2a445852c653774cf9f6af3038c6199730e
Oct 29 22:25:42.453159 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.453123    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:25:42.453242 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.453221    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:25:42.453286 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:42.453267    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:25:43.160485 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:43.160440    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt" event={"ID":"d166a89c-98b6-4209-b8d8-5b405090031b","Type":"ContainerStarted","Data":"be5122f9b627d20c476722007cd84519ee82eddd76a367ea193cecbbc4383e3d"}
Oct 29 22:25:43.161117 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:43.161080    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt" event={"ID":"d166a89c-98b6-4209-b8d8-5b405090031b","Type":"ContainerStarted","Data":"9e5bc8bba2f7f79b6026384d6c8cb2a445852c653774cf9f6af3038c6199730e"}
Oct 29 22:25:43.245071 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:43.241122    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt" podStartSLOduration=2.241099206 podStartE2EDuration="2.241099206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:43.220346717 +0000 UTC m=+682.131790274" watchObservedRunningTime="2026-10-29 22:25:43.241099206 +0000 UTC m=+682.152542836"
Oct 29 22:25:43.708007 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:25:43.707961    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:25:44.125382 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:44.125288    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:44.127020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:44.126994    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-m55mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 29 22:25:44.127146 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:44.127045    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt" podUID="d166a89c-98b6-4209-b8d8-5b405090031b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 29 22:25:45.125544 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:45.125498    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-m55mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 29 22:25:45.125955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:45.125562    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt" podUID="d166a89c-98b6-4209-b8d8-5b405090031b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 29 22:25:46.125162 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:46.125121    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-m55mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 29 22:25:46.125353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:46.125184    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt" podUID="d166a89c-98b6-4209-b8d8-5b405090031b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 29 22:25:47.126007 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:47.125968    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-m55mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 29 22:25:47.126583 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:47.126045    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt" podUID="d166a89c-98b6-4209-b8d8-5b405090031b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 29 22:25:48.125267 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:48.125230    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-m55mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 29 22:25:48.125530 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:48.125296    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt" podUID="d166a89c-98b6-4209-b8d8-5b405090031b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 29 22:25:49.126005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:49.125965    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-m55mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 29 22:25:49.126388 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:49.126052    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt" podUID="d166a89c-98b6-4209-b8d8-5b405090031b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 29 22:25:50.776606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:50.776569    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:50.777738 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:50.777713    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:50.778539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:50.778521    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt"
Oct 29 22:25:52.130625 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:25:52.130591    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"
Oct 29 22:26:07.465904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:07.465868    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"]
Oct 29 22:26:07.466491 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:07.466070    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd" podUID="d5b4e73d-dc80-4c28-8c48-d1248bc687b4" containerName="manager" containerID="cri-o://d4a5cc6889a0d0b661cc0a5df88268d05b20155e4b75c3f87729749061ce0552" gracePeriod=30
Oct 29 22:26:07.723180 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:07.723118    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"
Oct 29 22:26:07.800088 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:07.800051    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b4e73d-dc80-4c28-8c48-d1248bc687b4-kube-api-access-t2gtn\" (UniqueName: \"kubernetes.io/projected/d5b4e73d-dc80-4c28-8c48-d1248bc687b4-kube-api-access-t2gtn\") pod \"d5b4e73d-dc80-4c28-8c48-d1248bc687b4\" (UID: \"d5b4e73d-dc80-4c28-8c48-d1248bc687b4\") "
Oct 29 22:26:07.800285 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:07.800125    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5b4e73d-dc80-4c28-8c48-d1248bc687b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b4e73d-dc80-4c28-8c48-d1248bc687b4-metrics-certs\") pod \"d5b4e73d-dc80-4c28-8c48-d1248bc687b4\" (UID: \"d5b4e73d-dc80-4c28-8c48-d1248bc687b4\") "
Oct 29 22:26:07.802459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:07.802426    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b4e73d-dc80-4c28-8c48-d1248bc687b4-metrics-certs" pod "d5b4e73d-dc80-4c28-8c48-d1248bc687b4" (UID: "d5b4e73d-dc80-4c28-8c48-d1248bc687b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:26:07.802459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:07.802445    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b4e73d-dc80-4c28-8c48-d1248bc687b4-kube-api-access-t2gtn" pod "d5b4e73d-dc80-4c28-8c48-d1248bc687b4" (UID: "d5b4e73d-dc80-4c28-8c48-d1248bc687b4"). InnerVolumeSpecName "kube-api-access-t2gtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:26:07.901158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:07.901120    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b4e73d-dc80-4c28-8c48-d1248bc687b4-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:26:07.901158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:07.901149    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2gtn\" (UniqueName: \"kubernetes.io/projected/d5b4e73d-dc80-4c28-8c48-d1248bc687b4-kube-api-access-t2gtn\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:26:08.282618 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:08.282582    4559 generic.go:356] "Generic (PLEG): container finished" podID="d5b4e73d-dc80-4c28-8c48-d1248bc687b4" containerID="d4a5cc6889a0d0b661cc0a5df88268d05b20155e4b75c3f87729749061ce0552" exitCode=0
Oct 29 22:26:08.282789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:08.282631    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"
Oct 29 22:26:08.282789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:08.282637    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd" event={"ID":"d5b4e73d-dc80-4c28-8c48-d1248bc687b4","Type":"ContainerDied","Data":"d4a5cc6889a0d0b661cc0a5df88268d05b20155e4b75c3f87729749061ce0552"}
Oct 29 22:26:08.282789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:08.282678    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55b4c89cbd-6szvd" event={"ID":"d5b4e73d-dc80-4c28-8c48-d1248bc687b4","Type":"ContainerDied","Data":"4f93c3dc0a9a679bd41c6a34f1cbad93b2ccd62270bef0a8b9464ba6a1da8800"}
Oct 29 22:26:08.282789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:08.282700    4559 scope.go:122] "RemoveContainer" containerID="d4a5cc6889a0d0b661cc0a5df88268d05b20155e4b75c3f87729749061ce0552"
Oct 29 22:26:08.291611 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:08.291588    4559 scope.go:122] "RemoveContainer" containerID="d4a5cc6889a0d0b661cc0a5df88268d05b20155e4b75c3f87729749061ce0552"
Oct 29 22:26:08.291899 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:26:08.291879    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a5cc6889a0d0b661cc0a5df88268d05b20155e4b75c3f87729749061ce0552\": container with ID starting with d4a5cc6889a0d0b661cc0a5df88268d05b20155e4b75c3f87729749061ce0552 not found: ID does not exist" containerID="d4a5cc6889a0d0b661cc0a5df88268d05b20155e4b75c3f87729749061ce0552"
Oct 29 22:26:08.291968 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:08.291914    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a5cc6889a0d0b661cc0a5df88268d05b20155e4b75c3f87729749061ce0552"} err="failed to get container status \"d4a5cc6889a0d0b661cc0a5df88268d05b20155e4b75c3f87729749061ce0552\": rpc error: code = NotFound desc = could not find container \"d4a5cc6889a0d0b661cc0a5df88268d05b20155e4b75c3f87729749061ce0552\": container with ID starting with d4a5cc6889a0d0b661cc0a5df88268d05b20155e4b75c3f87729749061ce0552 not found: ID does not exist"
Oct 29 22:26:08.337049 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:08.337014    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"]
Oct 29 22:26:08.350545 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:08.350517    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-6szvd"]
Oct 29 22:26:09.226965 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:09.226933    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b4e73d-dc80-4c28-8c48-d1248bc687b4" path="/var/lib/kubelet/pods/d5b4e73d-dc80-4c28-8c48-d1248bc687b4/volumes"
Oct 29 22:26:13.693314 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:26:13.693274    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:26:34.713901 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.713859    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"]
Oct 29 22:26:34.714466 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.714451    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d5b4e73d-dc80-4c28-8c48-d1248bc687b4" containerName="manager"
Oct 29 22:26:34.714527 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.714468    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b4e73d-dc80-4c28-8c48-d1248bc687b4" containerName="manager"
Oct 29 22:26:34.714583 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.714572    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b4e73d-dc80-4c28-8c48-d1248bc687b4" containerName="manager"
Oct 29 22:26:34.721664 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.721635    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.753452 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.753418    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"]
Oct 29 22:26:34.753845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.753799    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-43b37d-openshift-default-dockercfg-g742w\""
Oct 29 22:26:34.838845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.838789    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"]
Oct 29 22:26:34.841410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.841376    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.841546 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.841429    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.841646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.841572    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.841646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.841615    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d2294b-e879-4782-b9e9-526f2aeecdba-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.841749 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.841707    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.841804 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.841742    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.841804 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.841769    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88mzx\" (UniqueName: \"kubernetes.io/projected/52d2294b-e879-4782-b9e9-526f2aeecdba-kube-api-access-88mzx\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.841924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.841803    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.842082 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.842040    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.843120 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.843099    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:34.866300 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.866266    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"]
Oct 29 22:26:34.908609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.908578    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-22d634-openshift-default-dockercfg-8sxjc\""
Oct 29 22:26:34.943131 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943090    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:34.943368 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943209    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.943368 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943281    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.943368 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943351    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:34.943543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943380    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggw4p\" (UniqueName: \"kubernetes.io/projected/0eb483d2-a79c-4e78-877e-d2a1f3285613-kube-api-access-ggw4p\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:34.943543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943411    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.943543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943517    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d2294b-e879-4782-b9e9-526f2aeecdba-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.943692 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943569    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eb483d2-a79c-4e78-877e-d2a1f3285613-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:34.943692 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943634    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:34.943692 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943680    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:34.943858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943718    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:34.943858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943757    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.943858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943761    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.943858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943829    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.943858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943842    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.943858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943850    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88mzx\" (UniqueName: \"kubernetes.io/projected/52d2294b-e879-4782-b9e9-526f2aeecdba-kube-api-access-88mzx\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.944164 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943900    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:34.944164 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.943957    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.944164 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.944018    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:34.944663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.944283    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d2294b-e879-4782-b9e9-526f2aeecdba-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.944663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.944332    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.944663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.944460    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.944663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.944625    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.946715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.946681    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.946715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.946704    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.965923 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.965850    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:34.966671 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:34.966643    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88mzx\" (UniqueName: \"kubernetes.io/projected/52d2294b-e879-4782-b9e9-526f2aeecdba-kube-api-access-88mzx\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:35.037957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.037910    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:35.045578 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.045552    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eb483d2-a79c-4e78-877e-d2a1f3285613-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.045728 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.045586    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.045728 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.045601    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.045728 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.045623    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.045728 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.045706    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.045955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.045736    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.045955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.045783    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.045955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.045912    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.045955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.045941    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggw4p\" (UniqueName: \"kubernetes.io/projected/0eb483d2-a79c-4e78-877e-d2a1f3285613-kube-api-access-ggw4p\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.046363 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.046339    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.046993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.046351    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eb483d2-a79c-4e78-877e-d2a1f3285613-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.047120 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.046442    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.047218 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.046748    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.047218 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.046960    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.048408 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.048382    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.049580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.049554    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.067911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.067879    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.068239 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.068219    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggw4p\" (UniqueName: \"kubernetes.io/projected/0eb483d2-a79c-4e78-877e-d2a1f3285613-kube-api-access-ggw4p\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.089879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.089557    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"]
Oct 29 22:26:35.095007 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.094983    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.122064 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.122025    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2ba10a-openshift-default-dockercfg-n9ldm\""
Oct 29 22:26:35.123900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.123873    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"]
Oct 29 22:26:35.168512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.168465    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:35.252485 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.252444    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.252656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.252504    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.252656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.252537    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.252656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.252571    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.252656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.252616    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8rrd\" (UniqueName: \"kubernetes.io/projected/1b88f5e2-5192-4da4-8f7c-51db3c907706-kube-api-access-h8rrd\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.252789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.252655    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.252789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.252678    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b88f5e2-5192-4da4-8f7c-51db3c907706-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.252789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.252763    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.252925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.252864    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.356490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.354083    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.356490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.354169    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b88f5e2-5192-4da4-8f7c-51db3c907706-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.356490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.354304    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.356490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.354437    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.356490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.354489    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.356490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.354519    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.356490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.354546    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.356490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.354580    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.356490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.354638    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8rrd\" (UniqueName: \"kubernetes.io/projected/1b88f5e2-5192-4da4-8f7c-51db3c907706-kube-api-access-h8rrd\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.356490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.355642    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.356490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.356116    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.356490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.356236    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.357238 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.356872    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b88f5e2-5192-4da4-8f7c-51db3c907706-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.357238 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.357170    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.359293 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.359262    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.359524 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.359506    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.378653 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.375351    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"]
Oct 29 22:26:35.381224 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.381190    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8rrd\" (UniqueName: \"kubernetes.io/projected/1b88f5e2-5192-4da4-8f7c-51db3c907706-kube-api-access-h8rrd\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.381486 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.381314    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"]
Oct 29 22:26:35.381486 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.381424    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.382616 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.382578    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.393057 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.393017    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:26:35.393156 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.393102    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:26:35.393200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.393173    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:26:35.408543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.398270    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2e046a-openshift-default-dockercfg-5g58t\""
Oct 29 22:26:35.409065 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.408912    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"]
Oct 29 22:26:35.422048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.421932    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67" event={"ID":"52d2294b-e879-4782-b9e9-526f2aeecdba","Type":"ContainerStarted","Data":"658098d36f116eb6db0a4ded0d37c2680e2fbbb9dfcfe1728bcae3e92758e428"}
Oct 29 22:26:35.430013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.429980    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:35.481621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.481380    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"]
Oct 29 22:26:35.484139 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:26:35.484109    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb483d2_a79c_4e78_877e_d2a1f3285613.slice/crio-2944595e062cfc7e9c7552e9b94d69b20e7ecf374cac5685ddcf467d35fa3d53 WatchSource:0}: Error finding container 2944595e062cfc7e9c7552e9b94d69b20e7ecf374cac5685ddcf467d35fa3d53: Status 404 returned error can't find the container with id 2944595e062cfc7e9c7552e9b94d69b20e7ecf374cac5685ddcf467d35fa3d53
Oct 29 22:26:35.487046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.486651    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:26:35.487046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.486714    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:26:35.487046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.486741    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:26:35.563727 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.557710    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.563727 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.557763    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.565925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.565417    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.565925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.565482    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.565925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.565533    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.565925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.565562    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.565925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.565603    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.565925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.565633    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.565925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.565657    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwmqs\" (UniqueName: \"kubernetes.io/projected/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-kube-api-access-lwmqs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.666587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.666532    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.666587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.666591    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.666868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.666629    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.666868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.666662    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.666868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.666691    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwmqs\" (UniqueName: \"kubernetes.io/projected/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-kube-api-access-lwmqs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.666868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.666864    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.667076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.666892    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.667076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.666959    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.667076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.667008    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.667677 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.667520    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.667789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.667672    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.667789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.667691    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.667995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.667975    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.668845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.668784    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.670770 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.670726    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.672066 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.672042    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.690750 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.690716    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.691640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.691616    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwmqs\" (UniqueName: \"kubernetes.io/projected/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-kube-api-access-lwmqs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.696482 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.696371    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:35.759978 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.759891    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"]
Oct 29 22:26:35.768538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.764634    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:26:35.768538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.764716    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:26:35.768538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:35.764760    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:26:36.059060 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:36.057256    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"]
Oct 29 22:26:36.060904 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:26:36.060866    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e5d770_5e34_4c1c_a5e9_3b62d070b61e.slice/crio-39af70143e96e8e9ccd7f627ff95b99320dac7ee0c9bdc04435e98899c7b95d0 WatchSource:0}: Error finding container 39af70143e96e8e9ccd7f627ff95b99320dac7ee0c9bdc04435e98899c7b95d0: Status 404 returned error can't find the container with id 39af70143e96e8e9ccd7f627ff95b99320dac7ee0c9bdc04435e98899c7b95d0
Oct 29 22:26:36.063630 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:36.063599    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:26:36.063729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:36.063665    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:26:36.063729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:36.063692    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:26:36.429689 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:36.429651    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp" event={"ID":"0eb483d2-a79c-4e78-877e-d2a1f3285613","Type":"ContainerStarted","Data":"b2c6caf9deb461614ae4687c49233327123a59e6c6a26fa1c689bd1410b6cb9e"}
Oct 29 22:26:36.429877 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:36.429716    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp" event={"ID":"0eb483d2-a79c-4e78-877e-d2a1f3285613","Type":"ContainerStarted","Data":"2944595e062cfc7e9c7552e9b94d69b20e7ecf374cac5685ddcf467d35fa3d53"}
Oct 29 22:26:36.437527 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:36.437488    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj" event={"ID":"10e5d770-5e34-4c1c-a5e9-3b62d070b61e","Type":"ContainerStarted","Data":"d9cfdee6327770ca5f7ec51add57b9ae09aef756952c8d72cb4731d40fd5a9c7"}
Oct 29 22:26:36.437687 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:36.437576    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj" event={"ID":"10e5d770-5e34-4c1c-a5e9-3b62d070b61e","Type":"ContainerStarted","Data":"39af70143e96e8e9ccd7f627ff95b99320dac7ee0c9bdc04435e98899c7b95d0"}
Oct 29 22:26:36.439198 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:36.439168    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" event={"ID":"1b88f5e2-5192-4da4-8f7c-51db3c907706","Type":"ContainerStarted","Data":"8a38048bad7695b501921c2f9341d014c3e649c9add4c1963958ab9812e34ae1"}
Oct 29 22:26:36.439303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:36.439215    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" event={"ID":"1b88f5e2-5192-4da4-8f7c-51db3c907706","Type":"ContainerStarted","Data":"4225214f87307f078aff41307650488f02a343ce01d9992dfb828315b6884f60"}
Oct 29 22:26:36.440714 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:36.440690    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67" event={"ID":"52d2294b-e879-4782-b9e9-526f2aeecdba","Type":"ContainerStarted","Data":"db9a82df3cf9c2a7fe143ba9bccb4fd7357a9d9dca1c66fe3b8636837fa1adc7"}
Oct 29 22:26:36.477351 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:36.477285    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp" podStartSLOduration=2.47726672 podStartE2EDuration="2.47726672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:36.475847071 +0000 UTC m=+735.387290626" watchObservedRunningTime="2026-10-29 22:26:36.47726672 +0000 UTC m=+735.388710275"
Oct 29 22:26:36.525299 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:36.525247    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" podStartSLOduration=1.525234038 podStartE2EDuration="1.525234038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:36.523122645 +0000 UTC m=+735.434566198" watchObservedRunningTime="2026-10-29 22:26:36.525234038 +0000 UTC m=+735.436677586"
Oct 29 22:26:36.568660 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:36.568603    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67" podStartSLOduration=2.568586266 podStartE2EDuration="2.568586266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:36.566661503 +0000 UTC m=+735.478105057" watchObservedRunningTime="2026-10-29 22:26:36.568586266 +0000 UTC m=+735.480029820"
Oct 29 22:26:36.611507 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:36.611443    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj" podStartSLOduration=1.611421027 podStartE2EDuration="1.611421027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:36.608706758 +0000 UTC m=+735.520150312" watchObservedRunningTime="2026-10-29 22:26:36.611421027 +0000 UTC m=+735.522864582"
Oct 29 22:26:36.697162 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:36.697078    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:37.038352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:37.038260    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:37.043565 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:37.043536    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:37.169648 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:37.169611    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:37.176008 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:37.175977    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:37.430958 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:37.430926    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:37.436363 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:37.436332    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:37.449185 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:37.449081    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:37.449185 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:37.449132    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:37.449185 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:37.449155    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:37.449756 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:37.449736    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:26:37.449983 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:37.449965    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:26:37.450204 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:37.450192    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:26:37.702352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:37.702269    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:38.454057 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:38.454018    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:38.454999 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:38.454979    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:26:43.697934 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:26:43.697897    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:26:44.908902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:44.908869    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"]
Oct 29 22:26:44.914000 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:44.913974    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"
Oct 29 22:26:44.938617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:44.938581    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"]
Oct 29 22:26:44.946774 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:44.946743    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-2ba10a-dockercfg-zv8lf\""
Oct 29 22:26:44.985252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:44.985211    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-2ba10a\""
Oct 29 22:26:45.092307 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.092253    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkdch\" (UniqueName: \"kubernetes.io/projected/c7b2c16e-ff49-4b9f-9e75-b0d2c4544532-kube-api-access-zkdch\") pod \"payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx\" (UID: \"c7b2c16e-ff49-4b9f-9e75-b0d2c4544532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"
Oct 29 22:26:45.092495 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.092419    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b2c16e-ff49-4b9f-9e75-b0d2c4544532-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx\" (UID: \"c7b2c16e-ff49-4b9f-9e75-b0d2c4544532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"
Oct 29 22:26:45.192969 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.192875    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b2c16e-ff49-4b9f-9e75-b0d2c4544532-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx\" (UID: \"c7b2c16e-ff49-4b9f-9e75-b0d2c4544532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"
Oct 29 22:26:45.192969 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.192937    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkdch\" (UniqueName: \"kubernetes.io/projected/c7b2c16e-ff49-4b9f-9e75-b0d2c4544532-kube-api-access-zkdch\") pod \"payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx\" (UID: \"c7b2c16e-ff49-4b9f-9e75-b0d2c4544532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"
Oct 29 22:26:45.193768 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.193739    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b2c16e-ff49-4b9f-9e75-b0d2c4544532-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx\" (UID: \"c7b2c16e-ff49-4b9f-9e75-b0d2c4544532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"
Oct 29 22:26:45.214154 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.214118    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkdch\" (UniqueName: \"kubernetes.io/projected/c7b2c16e-ff49-4b9f-9e75-b0d2c4544532-kube-api-access-zkdch\") pod \"payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx\" (UID: \"c7b2c16e-ff49-4b9f-9e75-b0d2c4544532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"
Oct 29 22:26:45.226511 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.226479    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"
Oct 29 22:26:45.527842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.524086    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:26:45.567835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.563232    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"]
Oct 29 22:26:45.583073 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.581310    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:26:45.794272 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.794193    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"]
Oct 29 22:26:45.799110 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.799083    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"
Oct 29 22:26:45.799551 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.799513    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q7kv\" (UniqueName: \"kubernetes.io/projected/ecd94d51-9423-41c3-8ab2-e94b52ab5904-kube-api-access-7q7kv\") pod \"payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj\" (UID: \"ecd94d51-9423-41c3-8ab2-e94b52ab5904\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"
Oct 29 22:26:45.799679 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.799662    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd94d51-9423-41c3-8ab2-e94b52ab5904-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj\" (UID: \"ecd94d51-9423-41c3-8ab2-e94b52ab5904\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"
Oct 29 22:26:45.818556 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.818520    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"]
Oct 29 22:26:45.823233 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.823206    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-08ebee-dockercfg-6tv86\""
Oct 29 22:26:45.829485 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.829461    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-08ebee\""
Oct 29 22:26:45.918427 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.918384    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q7kv\" (UniqueName: \"kubernetes.io/projected/ecd94d51-9423-41c3-8ab2-e94b52ab5904-kube-api-access-7q7kv\") pod \"payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj\" (UID: \"ecd94d51-9423-41c3-8ab2-e94b52ab5904\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"
Oct 29 22:26:45.919168 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.919126    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd94d51-9423-41c3-8ab2-e94b52ab5904-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj\" (UID: \"ecd94d51-9423-41c3-8ab2-e94b52ab5904\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"
Oct 29 22:26:45.924871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.920230    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd94d51-9423-41c3-8ab2-e94b52ab5904-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj\" (UID: \"ecd94d51-9423-41c3-8ab2-e94b52ab5904\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"
Oct 29 22:26:45.951760 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:45.951728    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q7kv\" (UniqueName: \"kubernetes.io/projected/ecd94d51-9423-41c3-8ab2-e94b52ab5904-kube-api-access-7q7kv\") pod \"payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj\" (UID: \"ecd94d51-9423-41c3-8ab2-e94b52ab5904\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"
Oct 29 22:26:46.112271 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:46.112189    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"
Oct 29 22:26:46.487490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:46.487459    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"]
Oct 29 22:26:46.489706 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:26:46.489677    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd94d51_9423_41c3_8ab2_e94b52ab5904.slice/crio-67a08e97ff8ee8a6d2154a16cc214cfece5f2424f5eed69ef16cf75f05882618 WatchSource:0}: Error finding container 67a08e97ff8ee8a6d2154a16cc214cfece5f2424f5eed69ef16cf75f05882618: Status 404 returned error can't find the container with id 67a08e97ff8ee8a6d2154a16cc214cfece5f2424f5eed69ef16cf75f05882618
Oct 29 22:26:46.497726 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:46.497697    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx" event={"ID":"c7b2c16e-ff49-4b9f-9e75-b0d2c4544532","Type":"ContainerStarted","Data":"7e455be50cd03620324893fba0d8291202763e6ac121c109957dc403305a28b4"}
Oct 29 22:26:46.851885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:46.849972    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"]
Oct 29 22:26:46.890085 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:46.880070    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"]
Oct 29 22:26:46.890085 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:46.880246    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"
Oct 29 22:26:46.922475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:46.921039    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-fc684b-dockercfg-qb7dp\""
Oct 29 22:26:46.922475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:46.921089    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-fc684b\""
Oct 29 22:26:46.929760 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:46.929731    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fb71c3f-8102-4f17-a698-5f8fc204b669-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s\" (UID: \"1fb71c3f-8102-4f17-a698-5f8fc204b669\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"
Oct 29 22:26:46.929936 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:46.929887    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dmfd\" (UniqueName: \"kubernetes.io/projected/1fb71c3f-8102-4f17-a698-5f8fc204b669-kube-api-access-9dmfd\") pod \"payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s\" (UID: \"1fb71c3f-8102-4f17-a698-5f8fc204b669\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"
Oct 29 22:26:47.031376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:47.031323    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fb71c3f-8102-4f17-a698-5f8fc204b669-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s\" (UID: \"1fb71c3f-8102-4f17-a698-5f8fc204b669\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"
Oct 29 22:26:47.031986 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:47.031754    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dmfd\" (UniqueName: \"kubernetes.io/projected/1fb71c3f-8102-4f17-a698-5f8fc204b669-kube-api-access-9dmfd\") pod \"payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s\" (UID: \"1fb71c3f-8102-4f17-a698-5f8fc204b669\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"
Oct 29 22:26:47.032344 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:47.032180    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fb71c3f-8102-4f17-a698-5f8fc204b669-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s\" (UID: \"1fb71c3f-8102-4f17-a698-5f8fc204b669\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"
Oct 29 22:26:47.055170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:47.055112    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dmfd\" (UniqueName: \"kubernetes.io/projected/1fb71c3f-8102-4f17-a698-5f8fc204b669-kube-api-access-9dmfd\") pod \"payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s\" (UID: \"1fb71c3f-8102-4f17-a698-5f8fc204b669\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"
Oct 29 22:26:47.247567 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:47.247534    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"
Oct 29 22:26:47.507332 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:47.507203    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj" event={"ID":"ecd94d51-9423-41c3-8ab2-e94b52ab5904","Type":"ContainerStarted","Data":"67a08e97ff8ee8a6d2154a16cc214cfece5f2424f5eed69ef16cf75f05882618"}
Oct 29 22:26:47.586332 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:47.586223    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"]
Oct 29 22:26:47.588748 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:26:47.588717    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb71c3f_8102_4f17_a698_5f8fc204b669.slice/crio-8e5f21bf987dddcf830c2733dd839eb635b0a85b507f5dfed81f4155200b80f8 WatchSource:0}: Error finding container 8e5f21bf987dddcf830c2733dd839eb635b0a85b507f5dfed81f4155200b80f8: Status 404 returned error can't find the container with id 8e5f21bf987dddcf830c2733dd839eb635b0a85b507f5dfed81f4155200b80f8
Oct 29 22:26:48.461794 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:48.461765    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"]
Oct 29 22:26:48.475601 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:48.475573    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"
Oct 29 22:26:48.488700 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:48.487239    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"]
Oct 29 22:26:48.498078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:48.497866    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-2e046a\""
Oct 29 22:26:48.498078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:48.498039    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-2e046a-dockercfg-pfzkk\""
Oct 29 22:26:48.579488 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:48.567440    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b67ef029-68f6-4d3e-acf1-6637d11c9ae2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7\" (UID: \"b67ef029-68f6-4d3e-acf1-6637d11c9ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"
Oct 29 22:26:48.579861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:48.579834    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vjdd\" (UniqueName: \"kubernetes.io/projected/b67ef029-68f6-4d3e-acf1-6637d11c9ae2-kube-api-access-4vjdd\") pod \"payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7\" (UID: \"b67ef029-68f6-4d3e-acf1-6637d11c9ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"
Oct 29 22:26:48.580145 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:48.578089    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s" event={"ID":"1fb71c3f-8102-4f17-a698-5f8fc204b669","Type":"ContainerStarted","Data":"8e5f21bf987dddcf830c2733dd839eb635b0a85b507f5dfed81f4155200b80f8"}
Oct 29 22:26:48.681068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:48.680973    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b67ef029-68f6-4d3e-acf1-6637d11c9ae2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7\" (UID: \"b67ef029-68f6-4d3e-acf1-6637d11c9ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"
Oct 29 22:26:48.681068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:48.681031    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vjdd\" (UniqueName: \"kubernetes.io/projected/b67ef029-68f6-4d3e-acf1-6637d11c9ae2-kube-api-access-4vjdd\") pod \"payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7\" (UID: \"b67ef029-68f6-4d3e-acf1-6637d11c9ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"
Oct 29 22:26:48.682366 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:48.682338    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b67ef029-68f6-4d3e-acf1-6637d11c9ae2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7\" (UID: \"b67ef029-68f6-4d3e-acf1-6637d11c9ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"
Oct 29 22:26:48.712701 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:48.709521    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vjdd\" (UniqueName: \"kubernetes.io/projected/b67ef029-68f6-4d3e-acf1-6637d11c9ae2-kube-api-access-4vjdd\") pod \"payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7\" (UID: \"b67ef029-68f6-4d3e-acf1-6637d11c9ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"
Oct 29 22:26:48.805380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:48.805339    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"
Oct 29 22:26:49.123670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:49.123636    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"]
Oct 29 22:26:49.125178 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:26:49.125150    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67ef029_68f6_4d3e_acf1_6637d11c9ae2.slice/crio-fe97ac6a16ccc138a56319125c4cb5ad51baeac65567218f2b43f92e2483a164 WatchSource:0}: Error finding container fe97ac6a16ccc138a56319125c4cb5ad51baeac65567218f2b43f92e2483a164: Status 404 returned error can't find the container with id fe97ac6a16ccc138a56319125c4cb5ad51baeac65567218f2b43f92e2483a164
Oct 29 22:26:49.588397 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:49.588130    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s" event={"ID":"1fb71c3f-8102-4f17-a698-5f8fc204b669","Type":"ContainerStarted","Data":"58191f3da6266783629a1a368bf35ab8410f6b73e0d880d69972b30346daa068"}
Oct 29 22:26:49.589278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:49.589192    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"
Oct 29 22:26:49.590332 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:49.590294    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx" event={"ID":"c7b2c16e-ff49-4b9f-9e75-b0d2c4544532","Type":"ContainerStarted","Data":"c2679dde734fc60ca409d39bda5f9fa652f7453dd299a0e52103a34ee7995826"}
Oct 29 22:26:49.590469 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:49.590353    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"
Oct 29 22:26:49.592224 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:49.592195    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj" event={"ID":"ecd94d51-9423-41c3-8ab2-e94b52ab5904","Type":"ContainerStarted","Data":"536623d37bcad3265f823d0d48ec46bb5096d871c44b055c3811335a8e0ac5fa"}
Oct 29 22:26:49.592340 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:49.592270    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"
Oct 29 22:26:49.594015 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:49.593992    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7" event={"ID":"b67ef029-68f6-4d3e-acf1-6637d11c9ae2","Type":"ContainerStarted","Data":"b083dc8de6a296a7eacd128265a4e015bcac2836569d28d0a70be40a3ba29603"}
Oct 29 22:26:49.594126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:49.594028    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7" event={"ID":"b67ef029-68f6-4d3e-acf1-6637d11c9ae2","Type":"ContainerStarted","Data":"fe97ac6a16ccc138a56319125c4cb5ad51baeac65567218f2b43f92e2483a164"}
Oct 29 22:26:49.594126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:49.594068    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"
Oct 29 22:26:49.637823 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:49.637061    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s" podStartSLOduration=2.050085086 podStartE2EDuration="3.637040278s" totalImagesPullingTime="1.586955192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:49.63487409 +0000 UTC m=+748.546317646" watchObservedRunningTime="2026-10-29 22:26:49.637040278 +0000 UTC m=+748.548483833"
Oct 29 22:26:49.697917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:49.697858    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj" podStartSLOduration=2.001971642 podStartE2EDuration="4.69783679s" totalImagesPullingTime="2.695865148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:49.696634157 +0000 UTC m=+748.608077712" watchObservedRunningTime="2026-10-29 22:26:49.69783679 +0000 UTC m=+748.609280345"
Oct 29 22:26:49.739723 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:49.739667    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7" podStartSLOduration=1.739652078 podStartE2EDuration="1.739652078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:49.737848961 +0000 UTC m=+748.649292517" watchObservedRunningTime="2026-10-29 22:26:49.739652078 +0000 UTC m=+748.651095632"
Oct 29 22:26:49.782594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:49.782540    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx" podStartSLOduration=2.954544393 podStartE2EDuration="5.782526908s" totalImagesPullingTime="2.827982515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:49.7813311 +0000 UTC m=+748.692774654" watchObservedRunningTime="2026-10-29 22:26:49.782526908 +0000 UTC m=+748.693970462"
Oct 29 22:26:50.596827 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:50.596784    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:26:51.437625 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.437588    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"]
Oct 29 22:26:51.441694 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.441671    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 22:26:51.474061 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.474024    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dzwhk\""
Oct 29 22:26:51.474242 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.474172    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:26:51.474302 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.474242    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 22:26:51.474357 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.474347    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:26:51.477973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.477940    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"]
Oct 29 22:26:51.478145 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.478072    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-43b37d-serving-cert\""
Oct 29 22:26:51.517319 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.517288    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e878d3a-382f-4f23-bc40-e28385b96f5e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-43b37d-c56588694-v4887\" (UID: \"6e878d3a-382f-4f23-bc40-e28385b96f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 22:26:51.517486 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.517439    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e878d3a-382f-4f23-bc40-e28385b96f5e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-43b37d-c56588694-v4887\" (UID: \"6e878d3a-382f-4f23-bc40-e28385b96f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 22:26:51.517565 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.517515    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e878d3a-382f-4f23-bc40-e28385b96f5e-metrics-certs\") pod \"maas-api-e2e-worker-w3-43b37d-c56588694-v4887\" (UID: \"6e878d3a-382f-4f23-bc40-e28385b96f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 22:26:51.517608 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.517568    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4z9p\" (UniqueName: \"kubernetes.io/projected/6e878d3a-382f-4f23-bc40-e28385b96f5e-kube-api-access-f4z9p\") pod \"maas-api-e2e-worker-w3-43b37d-c56588694-v4887\" (UID: \"6e878d3a-382f-4f23-bc40-e28385b96f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 22:26:51.618381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.618346    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e878d3a-382f-4f23-bc40-e28385b96f5e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-43b37d-c56588694-v4887\" (UID: \"6e878d3a-382f-4f23-bc40-e28385b96f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 22:26:51.618846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.618400    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e878d3a-382f-4f23-bc40-e28385b96f5e-metrics-certs\") pod \"maas-api-e2e-worker-w3-43b37d-c56588694-v4887\" (UID: \"6e878d3a-382f-4f23-bc40-e28385b96f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 22:26:51.618846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.618433    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4z9p\" (UniqueName: \"kubernetes.io/projected/6e878d3a-382f-4f23-bc40-e28385b96f5e-kube-api-access-f4z9p\") pod \"maas-api-e2e-worker-w3-43b37d-c56588694-v4887\" (UID: \"6e878d3a-382f-4f23-bc40-e28385b96f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 22:26:51.618846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.618468    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e878d3a-382f-4f23-bc40-e28385b96f5e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-43b37d-c56588694-v4887\" (UID: \"6e878d3a-382f-4f23-bc40-e28385b96f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 22:26:51.618846 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:26:51.618508    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-serving-cert: secret "maas-api-e2e-worker-w3-43b37d-serving-cert" not found
Oct 29 22:26:51.618846 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:26:51.618595    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e878d3a-382f-4f23-bc40-e28385b96f5e-maas-api-tls podName:6e878d3a-382f-4f23-bc40-e28385b96f5e nodeName:}" failed. No retries permitted until 2026-10-29 22:26:52.118573961 +0000 UTC m=+751.030017516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e878d3a-382f-4f23-bc40-e28385b96f5e-maas-api-tls") pod "maas-api-e2e-worker-w3-43b37d-c56588694-v4887" (UID: "6e878d3a-382f-4f23-bc40-e28385b96f5e") : secret "maas-api-e2e-worker-w3-43b37d-serving-cert" not found
Oct 29 22:26:51.619175 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.619156    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e878d3a-382f-4f23-bc40-e28385b96f5e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-43b37d-c56588694-v4887\" (UID: \"6e878d3a-382f-4f23-bc40-e28385b96f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 22:26:51.621189 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.621163    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e878d3a-382f-4f23-bc40-e28385b96f5e-metrics-certs\") pod \"maas-api-e2e-worker-w3-43b37d-c56588694-v4887\" (UID: \"6e878d3a-382f-4f23-bc40-e28385b96f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 22:26:51.646494 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:51.646457    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4z9p\" (UniqueName: \"kubernetes.io/projected/6e878d3a-382f-4f23-bc40-e28385b96f5e-kube-api-access-f4z9p\") pod \"maas-api-e2e-worker-w3-43b37d-c56588694-v4887\" (UID: \"6e878d3a-382f-4f23-bc40-e28385b96f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 22:26:52.125274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:52.125222    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e878d3a-382f-4f23-bc40-e28385b96f5e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-43b37d-c56588694-v4887\" (UID: \"6e878d3a-382f-4f23-bc40-e28385b96f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 22:26:52.127979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:52.127940    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e878d3a-382f-4f23-bc40-e28385b96f5e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-43b37d-c56588694-v4887\" (UID: \"6e878d3a-382f-4f23-bc40-e28385b96f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 22:26:52.358159 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:52.358118    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 22:26:52.665149 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:52.665121    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"]
Oct 29 22:26:52.667058 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:26:52.667024    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e878d3a_382f_4f23_bc40_e28385b96f5e.slice/crio-305f766acaa669cbb56dec126615089c16ff390cdc1445c54481aadcca1f48e4 WatchSource:0}: Error finding container 305f766acaa669cbb56dec126615089c16ff390cdc1445c54481aadcca1f48e4: Status 404 returned error can't find the container with id 305f766acaa669cbb56dec126615089c16ff390cdc1445c54481aadcca1f48e4
Oct 29 22:26:53.618985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:53.618911    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887" event={"ID":"6e878d3a-382f-4f23-bc40-e28385b96f5e","Type":"ContainerStarted","Data":"305f766acaa669cbb56dec126615089c16ff390cdc1445c54481aadcca1f48e4"}
Oct 29 22:26:56.523675 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.523598    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:26:56.700867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.694230    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"]
Oct 29 22:26:56.703659 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.703618    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887" event={"ID":"6e878d3a-382f-4f23-bc40-e28385b96f5e","Type":"ContainerStarted","Data":"a7aa7fc3c5473378a03c955d83388869885d9f3bcf4d4ae315b0ecb3801f9f8b"}
Oct 29 22:26:56.703862 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.703674    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 22:26:56.703862 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.703791    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 22:26:56.720731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.720677    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"]
Oct 29 22:26:56.730589 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.730558    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-08ebee-serving-cert\""
Oct 29 22:26:56.749652 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.749594    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb1155cc-6ac9-44d6-bd62-e09f88841686-maas-api-tls\") pod \"maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5\" (UID: \"bb1155cc-6ac9-44d6-bd62-e09f88841686\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 22:26:56.749943 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.749920    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb1155cc-6ac9-44d6-bd62-e09f88841686-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5\" (UID: \"bb1155cc-6ac9-44d6-bd62-e09f88841686\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 22:26:56.750114 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.750082    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb1155cc-6ac9-44d6-bd62-e09f88841686-metrics-certs\") pod \"maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5\" (UID: \"bb1155cc-6ac9-44d6-bd62-e09f88841686\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 22:26:56.750310 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.750297    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbtll\" (UniqueName: \"kubernetes.io/projected/bb1155cc-6ac9-44d6-bd62-e09f88841686-kube-api-access-hbtll\") pod \"maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5\" (UID: \"bb1155cc-6ac9-44d6-bd62-e09f88841686\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 22:26:56.825175 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.825075    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887" podStartSLOduration=2.270990286 podStartE2EDuration="5.825059828s" totalImagesPullingTime="3.554069542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:56.781956025 +0000 UTC m=+755.693399581" watchObservedRunningTime="2026-10-29 22:26:56.825059828 +0000 UTC m=+755.736503382"
Oct 29 22:26:56.861540 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.861494    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb1155cc-6ac9-44d6-bd62-e09f88841686-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5\" (UID: \"bb1155cc-6ac9-44d6-bd62-e09f88841686\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 22:26:56.862573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.862551    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb1155cc-6ac9-44d6-bd62-e09f88841686-metrics-certs\") pod \"maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5\" (UID: \"bb1155cc-6ac9-44d6-bd62-e09f88841686\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 22:26:56.862797 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.862783    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbtll\" (UniqueName: \"kubernetes.io/projected/bb1155cc-6ac9-44d6-bd62-e09f88841686-kube-api-access-hbtll\") pod \"maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5\" (UID: \"bb1155cc-6ac9-44d6-bd62-e09f88841686\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 22:26:56.863026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.863013    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb1155cc-6ac9-44d6-bd62-e09f88841686-maas-api-tls\") pod \"maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5\" (UID: \"bb1155cc-6ac9-44d6-bd62-e09f88841686\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 22:26:56.863231 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:26:56.863218    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-serving-cert: secret "maas-api-e2e-worker-w4-08ebee-serving-cert" not found
Oct 29 22:26:56.863351 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:26:56.863343    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb1155cc-6ac9-44d6-bd62-e09f88841686-maas-api-tls podName:bb1155cc-6ac9-44d6-bd62-e09f88841686 nodeName:}" failed. No retries permitted until 2026-10-29 22:26:57.363323386 +0000 UTC m=+756.274766935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bb1155cc-6ac9-44d6-bd62-e09f88841686-maas-api-tls") pod "maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5" (UID: "bb1155cc-6ac9-44d6-bd62-e09f88841686") : secret "maas-api-e2e-worker-w4-08ebee-serving-cert" not found
Oct 29 22:26:56.863570 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.862475    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb1155cc-6ac9-44d6-bd62-e09f88841686-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5\" (UID: \"bb1155cc-6ac9-44d6-bd62-e09f88841686\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 22:26:56.867485 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.867460    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb1155cc-6ac9-44d6-bd62-e09f88841686-metrics-certs\") pod \"maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5\" (UID: \"bb1155cc-6ac9-44d6-bd62-e09f88841686\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 22:26:56.886783 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:56.886751    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbtll\" (UniqueName: \"kubernetes.io/projected/bb1155cc-6ac9-44d6-bd62-e09f88841686-kube-api-access-hbtll\") pod \"maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5\" (UID: \"bb1155cc-6ac9-44d6-bd62-e09f88841686\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 22:26:57.370226 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:57.370189    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb1155cc-6ac9-44d6-bd62-e09f88841686-maas-api-tls\") pod \"maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5\" (UID: \"bb1155cc-6ac9-44d6-bd62-e09f88841686\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 22:26:57.372940 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:57.372903    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb1155cc-6ac9-44d6-bd62-e09f88841686-maas-api-tls\") pod \"maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5\" (UID: \"bb1155cc-6ac9-44d6-bd62-e09f88841686\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 22:26:57.623891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:57.619880    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 22:26:58.018975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:58.018950    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"]
Oct 29 22:26:58.021860 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:26:58.021747    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1155cc_6ac9_44d6_bd62_e09f88841686.slice/crio-b468aa38423ea132cbbd2c6c98b3ba66b5dc4c64930b2953440562c90e21a061 WatchSource:0}: Error finding container b468aa38423ea132cbbd2c6c98b3ba66b5dc4c64930b2953440562c90e21a061: Status 404 returned error can't find the container with id b468aa38423ea132cbbd2c6c98b3ba66b5dc4c64930b2953440562c90e21a061
Oct 29 22:26:58.734837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:58.733968    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5" event={"ID":"bb1155cc-6ac9-44d6-bd62-e09f88841686","Type":"ContainerStarted","Data":"b468aa38423ea132cbbd2c6c98b3ba66b5dc4c64930b2953440562c90e21a061"}
Oct 29 22:26:59.798842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:59.798350    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5" event={"ID":"bb1155cc-6ac9-44d6-bd62-e09f88841686","Type":"ContainerStarted","Data":"49752ad01da7019136dfe66e8acd03b398b13a84c9fd24bff33a3facf7983730"}
Oct 29 22:26:59.799577 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:59.799462    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 22:26:59.855834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:26:59.855614    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5" podStartSLOduration=3.354835515 podStartE2EDuration="3.855592282s" totalImagesPullingTime="500.756767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:59.853296106 +0000 UTC m=+758.764739689" watchObservedRunningTime="2026-10-29 22:26:59.855592282 +0000 UTC m=+758.767035837"
Oct 29 22:27:00.620347 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:00.620315    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"
Oct 29 22:27:00.621019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:00.620989    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"
Oct 29 22:27:00.621543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:00.621454    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"
Oct 29 22:27:00.622022 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:00.621998    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"
Oct 29 22:27:02.735337 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:02.735254    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 22:27:03.010559 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:03.010441    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:27:05.966700 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:05.966667    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 22:27:08.111409 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:08.111366    4559 generic.go:356] "Generic (PLEG): container finished" podID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" containerID="d9cfdee6327770ca5f7ec51add57b9ae09aef756952c8d72cb4731d40fd5a9c7" exitCode=0
Oct 29 22:27:08.111963 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:08.111438    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj" event={"ID":"10e5d770-5e34-4c1c-a5e9-3b62d070b61e","Type":"ContainerDied","Data":"d9cfdee6327770ca5f7ec51add57b9ae09aef756952c8d72cb4731d40fd5a9c7"}
Oct 29 22:27:08.111963 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:08.111803    4559 scope.go:122] "RemoveContainer" containerID="d9cfdee6327770ca5f7ec51add57b9ae09aef756952c8d72cb4731d40fd5a9c7"
Oct 29 22:27:08.114587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:08.114077    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:27:08.115376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:08.114875    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:27:08.115376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:08.114993    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:27:08.890767 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:08.454889    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:27:08.890767 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:08.489967    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:27:08.890767 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:08.489983    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:27:08.890767 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:08.490021    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67" podUID="52d2294b-e879-4782-b9e9-526f2aeecdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:27:08.890767 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:08.490025    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp" podUID="0eb483d2-a79c-4e78-877e-d2a1f3285613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:27:08.890767 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:08.489966    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:27:08.890767 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:08.490089    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:27:09.118785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:09.118750    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj" event={"ID":"10e5d770-5e34-4c1c-a5e9-3b62d070b61e","Type":"ContainerStarted","Data":"215a4eb15881c6d4dc5f3bb542ccb65fd627872244832d3b55193a74b230b323"}
Oct 29 22:27:09.697286 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:09.697254    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:27:10.058550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:09.698502    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 29 22:27:10.058550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:09.698548    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj" podUID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 29 22:27:10.126340 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:10.126307    4559 generic.go:356] "Generic (PLEG): container finished" podID="52d2294b-e879-4782-b9e9-526f2aeecdba" containerID="db9a82df3cf9c2a7fe143ba9bccb4fd7357a9d9dca1c66fe3b8636837fa1adc7" exitCode=0
Oct 29 22:27:10.126863 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:10.126389    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67" event={"ID":"52d2294b-e879-4782-b9e9-526f2aeecdba","Type":"ContainerDied","Data":"db9a82df3cf9c2a7fe143ba9bccb4fd7357a9d9dca1c66fe3b8636837fa1adc7"}
Oct 29 22:27:10.126927 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:10.126880    4559 scope.go:122] "RemoveContainer" containerID="db9a82df3cf9c2a7fe143ba9bccb4fd7357a9d9dca1c66fe3b8636837fa1adc7"
Oct 29 22:27:10.128256 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:10.128225    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:27:10.128361 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:10.128279    4559 generic.go:356] "Generic (PLEG): container finished" podID="0eb483d2-a79c-4e78-877e-d2a1f3285613" containerID="b2c6caf9deb461614ae4687c49233327123a59e6c6a26fa1c689bd1410b6cb9e" exitCode=0
Oct 29 22:27:10.128361 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:10.128311    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:27:10.128361 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:10.128346    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp" event={"ID":"0eb483d2-a79c-4e78-877e-d2a1f3285613","Type":"ContainerDied","Data":"b2c6caf9deb461614ae4687c49233327123a59e6c6a26fa1c689bd1410b6cb9e"}
Oct 29 22:27:10.128361 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:10.128356    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:27:10.128661 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:10.128648    4559 scope.go:122] "RemoveContainer" containerID="b2c6caf9deb461614ae4687c49233327123a59e6c6a26fa1c689bd1410b6cb9e"
Oct 29 22:27:10.129791 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:10.129762    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:27:10.129915 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:10.129861    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:27:10.129915 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:10.129895    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:27:10.594671 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:10.594624    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:27:10.697688 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:10.697214    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 29 22:27:10.697688 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:10.697286    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj" podUID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 29 22:27:11.188411 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:11.187418    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp" event={"ID":"0eb483d2-a79c-4e78-877e-d2a1f3285613","Type":"ContainerStarted","Data":"443e78d2c3f97f34d4b0e7c0db4f139cf15d17b89b4e57c92fd519df26e7ecd8"}
Oct 29 22:27:11.205157 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:11.205123    4559 generic.go:356] "Generic (PLEG): container finished" podID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerID="8a38048bad7695b501921c2f9341d014c3e649c9add4c1963958ab9812e34ae1" exitCode=0
Oct 29 22:27:11.205543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:11.205520    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" event={"ID":"1b88f5e2-5192-4da4-8f7c-51db3c907706","Type":"ContainerDied","Data":"8a38048bad7695b501921c2f9341d014c3e649c9add4c1963958ab9812e34ae1"}
Oct 29 22:27:11.206144 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:11.206122    4559 scope.go:122] "RemoveContainer" containerID="8a38048bad7695b501921c2f9341d014c3e649c9add4c1963958ab9812e34ae1"
Oct 29 22:27:11.210173 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:11.208735    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:27:11.210592 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:11.210488    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:27:11.210592 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:11.210553    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:27:11.214168 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:11.213728    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67" event={"ID":"52d2294b-e879-4782-b9e9-526f2aeecdba","Type":"ContainerStarted","Data":"ad591b2804aa639333022edfdeae1fbfb39cb05cb1935954c11abd6a61cd64e0"}
Oct 29 22:27:11.721664 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:11.721634    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 29 22:27:11.722488 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:11.722454    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj" podUID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 29 22:27:12.059889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:12.059588    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:27:12.059889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:12.059748    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 29 22:27:12.059889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:12.059793    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67" podUID="52d2294b-e879-4782-b9e9-526f2aeecdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 29 22:27:12.171238 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:12.171207    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:27:12.171578 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:12.171555    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 29 22:27:12.174921 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:12.174876    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp" podUID="0eb483d2-a79c-4e78-877e-d2a1f3285613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 29 22:27:12.424914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:12.420664    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" event={"ID":"1b88f5e2-5192-4da4-8f7c-51db3c907706","Type":"ContainerStarted","Data":"eecdbee41c5e6d5266d0f31e3c7e43f1769441df8f73e406f3d902d5b8c17f21"}
Oct 29 22:27:12.438896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:12.436419    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:27:12.446016 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:12.445984    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 29 22:27:12.446276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:12.446250    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 29 22:27:12.708001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:12.704579    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 29 22:27:12.708001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:12.704653    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj" podUID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 29 22:27:13.042780 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:13.042673    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 29 22:27:13.043105 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:13.043057    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67" podUID="52d2294b-e879-4782-b9e9-526f2aeecdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 29 22:27:13.179217 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:13.179180    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 29 22:27:13.179492 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:13.179465    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp" podUID="0eb483d2-a79c-4e78-877e-d2a1f3285613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 29 22:27:13.442013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:13.441971    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 29 22:27:13.442662 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:13.442624    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 29 22:27:13.700828 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:13.700301    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 29 22:27:13.700828 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:13.700362    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj" podUID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 29 22:27:13.789851 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:27:13.784278    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:27:14.044907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:14.040869    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 29 22:27:14.044907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:14.040927    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67" podUID="52d2294b-e879-4782-b9e9-526f2aeecdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 29 22:27:14.176843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:14.171193    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 29 22:27:14.176843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:14.171256    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp" podUID="0eb483d2-a79c-4e78-877e-d2a1f3285613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 29 22:27:14.456324 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:14.456292    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 29 22:27:14.456937 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:14.456908    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 29 22:27:14.698320 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:14.698221    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 29 22:27:14.698591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:14.698557    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj" podUID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 29 22:27:15.039434 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:15.039315    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 29 22:27:15.039434 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:15.039380    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67" podUID="52d2294b-e879-4782-b9e9-526f2aeecdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 29 22:27:15.172420 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:15.172384    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 29 22:27:15.173405 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:15.173368    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp" podUID="0eb483d2-a79c-4e78-877e-d2a1f3285613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 29 22:27:15.436247 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:15.436209    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 29 22:27:15.436944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:15.436521    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 29 22:27:15.721581 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:15.721225    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:27:15.723189 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:15.723165    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:27:15.724979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:15.724949    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:27:16.038886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:16.038310    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 29 22:27:16.038886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:16.038369    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67" podUID="52d2294b-e879-4782-b9e9-526f2aeecdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 29 22:27:16.170292 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:16.170253    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 29 22:27:16.170574 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:16.170542    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp" podUID="0eb483d2-a79c-4e78-877e-d2a1f3285613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 29 22:27:16.431152 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:16.431120    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 29 22:27:16.431420 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:16.431395    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 29 22:27:17.041841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:17.040917    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 29 22:27:17.041841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:17.040995    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67" podUID="52d2294b-e879-4782-b9e9-526f2aeecdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 29 22:27:17.172396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:17.172360    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 29 22:27:17.172722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:17.172678    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp" podUID="0eb483d2-a79c-4e78-877e-d2a1f3285613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 29 22:27:17.431097 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:17.431062    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 29 22:27:17.431320 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:17.431115    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 29 22:27:18.060838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:18.060717    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:27:18.062060 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:18.061935    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:27:18.062696 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:18.062669    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 22:27:18.173317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:18.173284    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:27:18.173631 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:18.173609    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:27:18.174779 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:18.174753    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:27:18.431220 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:18.431179    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 29 22:27:18.431395 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:18.431261    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 29 22:27:19.436032 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:19.435997    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:27:19.436533 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:19.436514    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:27:19.437152 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:19.437132    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:27:21.890611 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:21.890551    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"]
Oct 29 22:27:21.896560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:21.896536    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:21.926979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:21.926944    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-tmp-dir\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:21.927163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:21.927080    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-model-cache\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:21.927163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:21.927138    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-dshm\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:21.927290 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:21.927167    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154272dc-9da5-4f5d-ab1f-54282fe184d3-tls-certs\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:21.927290 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:21.927188    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-home\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:21.927401 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:21.927317    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-749dt\" (UniqueName: \"kubernetes.io/projected/154272dc-9da5-4f5d-ab1f-54282fe184d3-kube-api-access-749dt\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:21.927870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:21.927794    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2e046a\"/\"openshift-service-ca.crt\""
Oct 29 22:27:21.927870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:21.927845    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2e046a\"/\"default-dockercfg-ht7xw\""
Oct 29 22:27:21.928069 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:21.927794    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2e046a\"/\"kube-root-ca.crt\""
Oct 29 22:27:21.928069 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:21.927967    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2e046a\"/\"simulator-w0-2e046a-kserve-self-signed-certs\""
Oct 29 22:27:21.930132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:21.930108    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"]
Oct 29 22:27:22.029317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:22.029286    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-749dt\" (UniqueName: \"kubernetes.io/projected/154272dc-9da5-4f5d-ab1f-54282fe184d3-kube-api-access-749dt\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:22.029317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:22.029324    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-tmp-dir\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:22.029580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:22.029377    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-model-cache\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:22.029580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:22.029398    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-dshm\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:22.029580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:22.029415    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154272dc-9da5-4f5d-ab1f-54282fe184d3-tls-certs\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:22.029580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:22.029439    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-home\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:22.029886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:22.029860    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-model-cache\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:22.029973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:22.029864    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-tmp-dir\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:22.029973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:22.029955    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-home\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:22.032036 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:22.032013    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-dshm\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:22.032285 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:22.032267    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154272dc-9da5-4f5d-ab1f-54282fe184d3-tls-certs\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:22.051720 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:22.051691    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-749dt\" (UniqueName: \"kubernetes.io/projected/154272dc-9da5-4f5d-ab1f-54282fe184d3-kube-api-access-749dt\") pod \"simulator-w0-2e046a-kserve-7df67475d7-x79mn\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") " pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:22.083866 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:22.083823    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:27:22.211550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:22.211450    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:22.526797 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:27:22.526754    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod154272dc_9da5_4f5d_ab1f_54282fe184d3.slice/crio-6ee10e73f7ab12f7f8a2614f6be0a6e6fdfce3b6883a8021626a2a4ec0cc7309 WatchSource:0}: Error finding container 6ee10e73f7ab12f7f8a2614f6be0a6e6fdfce3b6883a8021626a2a4ec0cc7309: Status 404 returned error can't find the container with id 6ee10e73f7ab12f7f8a2614f6be0a6e6fdfce3b6883a8021626a2a4ec0cc7309
Oct 29 22:27:22.530471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:22.530436    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"]
Oct 29 22:27:22.540686 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:22.540639    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" event={"ID":"154272dc-9da5-4f5d-ab1f-54282fe184d3","Type":"ContainerStarted","Data":"6ee10e73f7ab12f7f8a2614f6be0a6e6fdfce3b6883a8021626a2a4ec0cc7309"}
Oct 29 22:27:23.548024 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:23.547990    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" event={"ID":"154272dc-9da5-4f5d-ab1f-54282fe184d3","Type":"ContainerStarted","Data":"4e618b766aed8eea4c86ec3fbb1c2d27922409f926fc50a154a074c3b62014da"}
Oct 29 22:27:23.590182 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:23.590122    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podStartSLOduration=2.590105 podStartE2EDuration="2.590105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:23.588758082 +0000 UTC m=+782.500201638" watchObservedRunningTime="2026-10-29 22:27:23.590105 +0000 UTC m=+782.501548553"
Oct 29 22:27:24.212022 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:24.211984    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:27.790783 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:27.790741    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"]
Oct 29 22:27:27.795033 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:27.795005    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:27.825152 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:27.825108    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2ba10a\"/\"openshift-service-ca.crt\""
Oct 29 22:27:27.825152 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:27.825143    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2ba10a\"/\"kube-root-ca.crt\""
Oct 29 22:27:27.825424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:27.825207    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2ba10a\"/\"simulator-w2-2ba10a-kserve-self-signed-certs\""
Oct 29 22:27:27.825424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:27.825279    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2ba10a\"/\"default-dockercfg-56tcf\""
Oct 29 22:27:27.828827 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:27.828787    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"]
Oct 29 22:27:27.899131 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:27.899097    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkmzv\" (UniqueName: \"kubernetes.io/projected/6d535825-4bca-4a43-8e4e-4fcae50adffa-kube-api-access-pkmzv\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:27.899323 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:27.899150    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-model-cache\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:27.899323 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:27.899248    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-dshm\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:27.899323 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:27.899286    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-home\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:27.899323 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:27.899312    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-tmp-dir\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:27.899463 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:27.899448    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d535825-4bca-4a43-8e4e-4fcae50adffa-tls-certs\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:28.000339 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:28.000304    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-model-cache\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:28.000529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:28.000359    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-dshm\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:28.000529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:28.000391    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-home\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:28.000529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:28.000418    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-tmp-dir\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:28.000529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:28.000493    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d535825-4bca-4a43-8e4e-4fcae50adffa-tls-certs\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:28.000755 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:28.000540    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkmzv\" (UniqueName: \"kubernetes.io/projected/6d535825-4bca-4a43-8e4e-4fcae50adffa-kube-api-access-pkmzv\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:28.000891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:28.000862    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-model-cache\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:28.000970 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:28.000893    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-home\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:28.001036 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:28.001015    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-tmp-dir\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:28.002843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:28.002789    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-dshm\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:28.003161 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:28.003142    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d535825-4bca-4a43-8e4e-4fcae50adffa-tls-certs\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:28.034184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:28.034110    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkmzv\" (UniqueName: \"kubernetes.io/projected/6d535825-4bca-4a43-8e4e-4fcae50adffa-kube-api-access-pkmzv\") pod \"simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") " pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:28.109656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:28.109568    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:28.416074 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:28.416042    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"]
Oct 29 22:27:28.416849 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:27:28.416803    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d535825_4bca_4a43_8e4e_4fcae50adffa.slice/crio-0829983355d86de87e6037fb723811e8f4473148dba74ebb3605c71048b014d4 WatchSource:0}: Error finding container 0829983355d86de87e6037fb723811e8f4473148dba74ebb3605c71048b014d4: Status 404 returned error can't find the container with id 0829983355d86de87e6037fb723811e8f4473148dba74ebb3605c71048b014d4
Oct 29 22:27:28.599913 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:28.599874    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" event={"ID":"6d535825-4bca-4a43-8e4e-4fcae50adffa","Type":"ContainerStarted","Data":"91d64a1cfcfd9dabeb3fa3e22dd7e3ab6069ff575a7b828c44a5ab24d211c2e9"}
Oct 29 22:27:28.600102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:28.599927    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" event={"ID":"6d535825-4bca-4a43-8e4e-4fcae50adffa","Type":"ContainerStarted","Data":"0829983355d86de87e6037fb723811e8f4473148dba74ebb3605c71048b014d4"}
Oct 29 22:27:28.644970 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:28.644907    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podStartSLOduration=1.644889027 podStartE2EDuration="1.644889027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:28.644345974 +0000 UTC m=+787.555789530" watchObservedRunningTime="2026-10-29 22:27:28.644889027 +0000 UTC m=+787.556332609"
Oct 29 22:27:29.109933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:29.109899    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:32.212561 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:32.212507    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:32.225222 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:32.225194    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:32.632081 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:32.631994    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:27:35.501102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:35.501066    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:27:38.110378 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:38.110342    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:38.123550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:38.123514    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:38.667218 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:38.667189    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:27:39.282497 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.282455    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"]
Oct 29 22:27:39.286107 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.286081    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.306523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.306473    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-784e7a\"/\"openshift-service-ca.crt\""
Oct 29 22:27:39.306712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.306601    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-784e7a\"/\"kube-root-ca.crt\""
Oct 29 22:27:39.318487 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.318457    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-784e7a\"/\"default-dockercfg-d8kph\""
Oct 29 22:27:39.318716 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.318562    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-784e7a\"/\"premium-w6-784e7a-kserve-self-signed-certs\""
Oct 29 22:27:39.322019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.321982    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"]
Oct 29 22:27:39.349865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.349825    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-model-cache\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.349865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.349873    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-tls-certs\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.350121 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.349902    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-tmp-dir\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.350121 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.350074    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-home\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.350239 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.350135    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b75zm\" (UniqueName: \"kubernetes.io/projected/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-kube-api-access-b75zm\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.350239 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.350212    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-dshm\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.450927 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.450878    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-home\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.451122 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.450942    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b75zm\" (UniqueName: \"kubernetes.io/projected/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-kube-api-access-b75zm\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.451122 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.451024    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-dshm\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.451122 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.451114    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-model-cache\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.451301 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.451152    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-tls-certs\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.451301 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.451191    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-tmp-dir\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.451427 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.451397    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-home\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.451862 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.451785    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-tmp-dir\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.451998 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.451893    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-model-cache\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.453886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.453840    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-dshm\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.455172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.454961    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-tls-certs\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.471356 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.471325    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b75zm\" (UniqueName: \"kubernetes.io/projected/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-kube-api-access-b75zm\") pod \"premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") " pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.601835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.601720    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:39.910739 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:39.910710    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"]
Oct 29 22:27:39.911931 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:27:39.911898    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96d9ca2d_e01b_4bf4_8cfc_2d0fe145e51f.slice/crio-47e08b23930ff5378083e650b8b3c3010ecf4190458275d3326b80d97b29f957 WatchSource:0}: Error finding container 47e08b23930ff5378083e650b8b3c3010ecf4190458275d3326b80d97b29f957: Status 404 returned error can't find the container with id 47e08b23930ff5378083e650b8b3c3010ecf4190458275d3326b80d97b29f957
Oct 29 22:27:40.673308 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:40.673273    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" event={"ID":"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f","Type":"ContainerStarted","Data":"179b810267495e341681e03620615bf5cf1d3cd62b700ab96f85be85de5fe8d6"}
Oct 29 22:27:40.673308 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:40.673316    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" event={"ID":"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f","Type":"ContainerStarted","Data":"47e08b23930ff5378083e650b8b3c3010ecf4190458275d3326b80d97b29f957"}
Oct 29 22:27:40.714143 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:40.714085    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podStartSLOduration=1.714069075 podStartE2EDuration="1.714069075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:40.713416074 +0000 UTC m=+799.624859628" watchObservedRunningTime="2026-10-29 22:27:40.714069075 +0000 UTC m=+799.625512630"
Oct 29 22:27:41.602566 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:41.602535    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:43.695150 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:27:43.695116    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:27:48.218205 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:48.218163    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:27:49.601873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:49.601839    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:49.615370 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:49.615342    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:49.742265 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:49.742235    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:27:59.099700 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.099661    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"]
Oct 29 22:27:59.105198 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.105164    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.129873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.129841    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"]
Oct 29 22:27:59.139965 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.139931    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-43b37d\"/\"premium-w3-43b37d-kserve-self-signed-certs\""
Oct 29 22:27:59.140124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.140057    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-43b37d\"/\"openshift-service-ca.crt\""
Oct 29 22:27:59.140185 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.140130    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-43b37d\"/\"kube-root-ca.crt\""
Oct 29 22:27:59.186086 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.186046    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-dshm\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.186275 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.186119    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-home\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.186275 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.186212    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-tls-certs\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.186275 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.186261    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-model-cache\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.186436 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.186291    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-tmp-dir\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.186497 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.186433    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgs49\" (UniqueName: \"kubernetes.io/projected/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-kube-api-access-xgs49\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.223606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.223578    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-43b37d\"/\"default-dockercfg-k2fzk\""
Oct 29 22:27:59.287888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.287798    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-model-cache\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.288068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.287900    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-tmp-dir\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.288068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.287945    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgs49\" (UniqueName: \"kubernetes.io/projected/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-kube-api-access-xgs49\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.288068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.287976    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-dshm\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.288068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.288036    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-home\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.288288 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.288109    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-tls-certs\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.288288 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.288268    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-tmp-dir\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.288404 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.288280    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-model-cache\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.288493 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.288470    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-home\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.290394 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.290369    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-dshm\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.290627 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.290610    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-tls-certs\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.339259 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.324300    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgs49\" (UniqueName: \"kubernetes.io/projected/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-kube-api-access-xgs49\") pod \"premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") " pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.426468 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.426434    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:27:59.735727 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:27:59.735688    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53116ca8_8cee_4efb_8f3f_4ccb571ff7ca.slice/crio-0ba376eae89d97bb53eee4ac541375d8c3f46eb662e34233537b917da8678770 WatchSource:0}: Error finding container 0ba376eae89d97bb53eee4ac541375d8c3f46eb662e34233537b917da8678770: Status 404 returned error can't find the container with id 0ba376eae89d97bb53eee4ac541375d8c3f46eb662e34233537b917da8678770
Oct 29 22:27:59.736753 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.736723    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"]
Oct 29 22:27:59.785433 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:27:59.785398    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" event={"ID":"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca","Type":"ContainerStarted","Data":"0ba376eae89d97bb53eee4ac541375d8c3f46eb662e34233537b917da8678770"}
Oct 29 22:28:00.793201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:00.793169    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" event={"ID":"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca","Type":"ContainerStarted","Data":"b002d69b20b8a3b2f073462fcdc8d08ca0fac491ac93d3f4f477ef04243130f0"}
Oct 29 22:28:00.839231 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:00.839175    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podStartSLOduration=1.839159818 podStartE2EDuration="1.839159818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:00.837285916 +0000 UTC m=+819.748729472" watchObservedRunningTime="2026-10-29 22:28:00.839159818 +0000 UTC m=+819.750603427"
Oct 29 22:28:01.427302 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:01.427272    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:28:03.197271 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:03.197237    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:28:09.427147 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:09.427110    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:28:09.440106 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:09.440078    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:28:09.854416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:09.854330    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:28:13.695510 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:28:13.695474    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:28:15.345226 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.345182    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"]
Oct 29 22:28:15.349249 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.349225    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.380723 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.380693    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-08ebee\"/\"kube-root-ca.crt\""
Oct 29 22:28:15.380922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.380787    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-08ebee\"/\"default-dockercfg-pfpfc\""
Oct 29 22:28:15.380922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.380694    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-08ebee\"/\"openshift-service-ca.crt\""
Oct 29 22:28:15.383358 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.383334    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-08ebee\"/\"premium-w4-08ebee-kserve-self-signed-certs\""
Oct 29 22:28:15.383880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.383854    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"]
Oct 29 22:28:15.492415 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.492374    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-model-cache\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.492611 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.492432    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sznh9\" (UniqueName: \"kubernetes.io/projected/779ec9d1-52de-4f60-9f5e-4be62ceb311f-kube-api-access-sznh9\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.492611 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.492582    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779ec9d1-52de-4f60-9f5e-4be62ceb311f-tls-certs\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.492709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.492686    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-home\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.492774 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.492760    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-dshm\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.492966 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.492942    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-tmp-dir\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.598518 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.598430    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-tmp-dir\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.598518 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.598478    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-model-cache\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.598518 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.598502    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sznh9\" (UniqueName: \"kubernetes.io/projected/779ec9d1-52de-4f60-9f5e-4be62ceb311f-kube-api-access-sznh9\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.598872 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.598541    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779ec9d1-52de-4f60-9f5e-4be62ceb311f-tls-certs\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.598872 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.598601    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-home\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.598872 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.598638    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-dshm\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.599772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.599744    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-tmp-dir\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.600007 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.599825    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-model-cache\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.600114 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.599843    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-home\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.621297 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.615322    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779ec9d1-52de-4f60-9f5e-4be62ceb311f-tls-certs\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.622490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.622460    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-dshm\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.637847 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.634577    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sznh9\" (UniqueName: \"kubernetes.io/projected/779ec9d1-52de-4f60-9f5e-4be62ceb311f-kube-api-access-sznh9\") pod \"premium-w4-08ebee-kserve-7b64956b4f-8x224\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") " pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.663751 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.663718    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:15.977670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:15.977633    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"]
Oct 29 22:28:15.984127 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:28:15.984078    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod779ec9d1_52de_4f60_9f5e_4be62ceb311f.slice/crio-ac630464464f942234be329f4fd675f9c33ad7ed0202c400207601706756ecdc WatchSource:0}: Error finding container ac630464464f942234be329f4fd675f9c33ad7ed0202c400207601706756ecdc: Status 404 returned error can't find the container with id ac630464464f942234be329f4fd675f9c33ad7ed0202c400207601706756ecdc
Oct 29 22:28:16.888670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:16.888623    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" event={"ID":"779ec9d1-52de-4f60-9f5e-4be62ceb311f","Type":"ContainerStarted","Data":"c14e601cf13401fd733920ccd520cd01cca892f63c229a14cb50b21b4cad9afb"}
Oct 29 22:28:16.889217 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:16.888682    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" event={"ID":"779ec9d1-52de-4f60-9f5e-4be62ceb311f","Type":"ContainerStarted","Data":"ac630464464f942234be329f4fd675f9c33ad7ed0202c400207601706756ecdc"}
Oct 29 22:28:16.937829 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:16.937762    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podStartSLOduration=1.937744288 podStartE2EDuration="1.937744288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:16.935481905 +0000 UTC m=+835.846925460" watchObservedRunningTime="2026-10-29 22:28:16.937744288 +0000 UTC m=+835.849187842"
Oct 29 22:28:17.664099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:17.664058    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:25.664227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:25.664181    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:25.677735 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:25.677706    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:25.971939 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:25.971842    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:28:27.342659 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:27.342615    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:28:35.299433 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.299350    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"]
Oct 29 22:28:35.304510 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.304488    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.323313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.323286    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-fc684b-openshift-default-dockercfg-8hgq7\""
Oct 29 22:28:35.328945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.328916    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.329078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.328984    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.329078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.329057    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.329167 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.329131    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.329227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.329207    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.329264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.329241    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.329317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.329281    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.329413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.329397    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.329475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.329430    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz2pb\" (UniqueName: \"kubernetes.io/projected/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-kube-api-access-vz2pb\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.337972 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.337942    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"]
Oct 29 22:28:35.430432 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.430391    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.430633 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.430449    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.430633 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.430503    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.430633 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.430531    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.430633 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.430580    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.430886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.430640    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.430886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.430666    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz2pb\" (UniqueName: \"kubernetes.io/projected/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-kube-api-access-vz2pb\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.430886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.430739    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.430886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.430804    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.431102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.430886    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.431102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.430841    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.431102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.430917    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.431258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.431216    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.431440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.431420    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.433080 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.433056    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.433336 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.433315    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.455491 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.455463    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.460147 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.460124    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz2pb\" (UniqueName: \"kubernetes.io/projected/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-kube-api-access-vz2pb\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.618139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.618050    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:35.966751 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.966717    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"]
Oct 29 22:28:35.983489 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:28:35.983379    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19ddc8f_507a_4427_91e9_12f4c0fb1ac9.slice/crio-7266973d44bc1c45556a258d4b7d76118ddcde539ffd4e09bb220a26a7787a15 WatchSource:0}: Error finding container 7266973d44bc1c45556a258d4b7d76118ddcde539ffd4e09bb220a26a7787a15: Status 404 returned error can't find the container with id 7266973d44bc1c45556a258d4b7d76118ddcde539ffd4e09bb220a26a7787a15
Oct 29 22:28:35.986427 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.986392    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:28:35.986548 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.986496    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:28:35.986616 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:35.986559    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:28:36.026033 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:36.025992    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k" event={"ID":"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9","Type":"ContainerStarted","Data":"7266973d44bc1c45556a258d4b7d76118ddcde539ffd4e09bb220a26a7787a15"}
Oct 29 22:28:37.091568 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:37.090630    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k" event={"ID":"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9","Type":"ContainerStarted","Data":"5cbae560fc3b9d7a526036a9cfa29a6c4646e9245e1ab0bbf2b8092816f5289c"}
Oct 29 22:28:37.168841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:37.166758    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k" podStartSLOduration=2.166734999 podStartE2EDuration="2.166734999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:37.150280233 +0000 UTC m=+856.061723816" watchObservedRunningTime="2026-10-29 22:28:37.166734999 +0000 UTC m=+856.078178554"
Oct 29 22:28:37.619190 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:37.619151    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:37.621066 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:37.621037    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:28:37.621203 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:37.621120    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k" podUID="f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 22:28:38.618588 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:38.618555    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:28:38.619033 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:38.618612    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k" podUID="f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 22:28:39.619248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:39.619215    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:28:39.619725 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:39.619270    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k" podUID="f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 22:28:40.619138 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:40.619100    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:28:40.619412 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:40.619164    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k" podUID="f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 22:28:41.619161 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:41.619127    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:28:41.619382 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:41.619182    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k" podUID="f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 22:28:42.618945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:42.618902    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:28:42.619359 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:42.618983    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k" podUID="f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 22:28:43.624128 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:43.624088    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:43.624554 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:43.624476    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:43.625346 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:43.625314    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 22:28:43.698376 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:28:43.698332    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:28:49.927872 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:49.927834    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"]
Oct 29 22:28:49.932091 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:49.932071    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:49.947875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:49.947845    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-784e7a-openshift-default-dockercfg-hn6qv\""
Oct 29 22:28:49.964396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:49.964362    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"]
Oct 29 22:28:49.976072 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:49.976029    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:49.976278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:49.976121    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:49.976278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:49.976164    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fk95\" (UniqueName: \"kubernetes.io/projected/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-kube-api-access-6fk95\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:49.976278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:49.976221    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:49.976451 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:49.976303    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:49.976451 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:49.976341    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:49.976451 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:49.976393    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:49.976451 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:49.976442    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:49.976657 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:49.976530    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.078083 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.078036    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.078268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.078117    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fk95\" (UniqueName: \"kubernetes.io/projected/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-kube-api-access-6fk95\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.078268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.078150    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.078268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.078210    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.078268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.078241    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.078479 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.078291    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.078479 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.078348    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.078479 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.078398    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.078479 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.078462    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.078679 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.078478    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.078781 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.078756    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.079053 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.078848    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.079193 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.079171    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.079719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.079694    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.081400 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.081377    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.081480 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.081407    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.099078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.099041    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fk95\" (UniqueName: \"kubernetes.io/projected/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-kube-api-access-6fk95\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.099233 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.099215    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.247500 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.247407    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:50.269262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.268836    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:28:50.581102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.581074    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"]
Oct 29 22:28:50.582163 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:28:50.582135    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20009952_bbfe_46ae_8ce8_1fbee6b3c8a3.slice/crio-525ac70383b84c6c154a22d0962c22c27a767f869d558bffe3bec0ffc47a49dd WatchSource:0}: Error finding container 525ac70383b84c6c154a22d0962c22c27a767f869d558bffe3bec0ffc47a49dd: Status 404 returned error can't find the container with id 525ac70383b84c6c154a22d0962c22c27a767f869d558bffe3bec0ffc47a49dd
Oct 29 22:28:50.584878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.584795    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:28:50.584971 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.584930    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:28:50.585017 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.584971    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:28:50.621749 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.621721    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"]
Oct 29 22:28:50.627075 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.627050    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.642490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.642456    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-08ebee-openshift-default-dockercfg-4sgmc\""
Oct 29 22:28:50.653425 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.653393    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"]
Oct 29 22:28:50.685310 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.685261    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.685884 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.685800    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.686258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.686236    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.686540 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.686503    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.686888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.686854    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.687089 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.687072    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.687180 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.687105    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q8rw\" (UniqueName: \"kubernetes.io/projected/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-kube-api-access-2q8rw\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.687180 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.687160    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.687355 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.687189    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.790168 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.790123    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.790376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.790213    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.790376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.790268    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.790376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.790317    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.790572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.790390    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.790572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.790438    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.790572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.790465    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q8rw\" (UniqueName: \"kubernetes.io/projected/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-kube-api-access-2q8rw\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.790572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.790504    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.790572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.790531    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.791618 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.791533    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.792615 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.792581    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.792792 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.792608    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.792933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.792867    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.793254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.793237    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.796040 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.795944    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.797104 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.797079    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.811928 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.811900    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q8rw\" (UniqueName: \"kubernetes.io/projected/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-kube-api-access-2q8rw\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.815282 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.815256    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:50.940983 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:50.940953    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:51.206890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:51.206288    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw" event={"ID":"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3","Type":"ContainerStarted","Data":"deac0201b3c242632a856d6ce23e0a8b86b0a9b2d5bdb8efdbb4398c65886945"}
Oct 29 22:28:51.206890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:51.206366    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw" event={"ID":"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3","Type":"ContainerStarted","Data":"525ac70383b84c6c154a22d0962c22c27a767f869d558bffe3bec0ffc47a49dd"}
Oct 29 22:28:51.331050 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:51.328354    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:28:51.331050 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:51.328415    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw" podUID="20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 29 22:28:51.339751 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:51.339683    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw" podStartSLOduration=2.339663657 podStartE2EDuration="2.339663657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:51.328997122 +0000 UTC m=+870.240440707" watchObservedRunningTime="2026-10-29 22:28:51.339663657 +0000 UTC m=+870.251107215"
Oct 29 22:28:51.361979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:51.361938    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:28:51.362300 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:51.362257    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:28:51.362471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:51.362453    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:28:51.392554 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:51.392523    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"]
Oct 29 22:28:51.392734 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:51.392590    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:52.282205 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:52.282167    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:28:52.282829 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:52.282772    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw" podUID="20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 29 22:28:52.298285 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:52.298250    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn" event={"ID":"e3bc3b1a-f61c-4835-98e5-b9efccd4470e","Type":"ContainerStarted","Data":"eb4683ce3c129c13ab9af10b5e8a3b00f6523e7c95fc7a97bb5100ad45a54a3e"}
Oct 29 22:28:52.299744 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:52.299714    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn" event={"ID":"e3bc3b1a-f61c-4835-98e5-b9efccd4470e","Type":"ContainerStarted","Data":"67848abb75153432414e4774ae7884d15e5b89ab1da00a82ca893f4a7f7c2aae"}
Oct 29 22:28:52.351989 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:52.348787    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn" podStartSLOduration=2.348764994 podStartE2EDuration="2.348764994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:52.346584157 +0000 UTC m=+871.258027723" watchObservedRunningTime="2026-10-29 22:28:52.348764994 +0000 UTC m=+871.260208550"
Oct 29 22:28:52.959836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:52.955879    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:52.959836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:52.956721    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:28:52.959836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:52.956776    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn" podUID="e3bc3b1a-f61c-4835-98e5-b9efccd4470e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 29 22:28:53.251298 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:53.251199    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:28:53.251563 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:53.251533    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw" podUID="20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 29 22:28:53.954835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:53.948247    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:28:53.954835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:53.948315    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn" podUID="e3bc3b1a-f61c-4835-98e5-b9efccd4470e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 29 22:28:54.254894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.254266    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:28:54.254894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.254330    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw" podUID="20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 29 22:28:54.599729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.599585    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"]
Oct 29 22:28:54.603826 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.603784    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"
Oct 29 22:28:54.621712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.621676    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:28:54.663842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.659404    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:28:54.709034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.708936    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"]
Oct 29 22:28:54.733853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.721520    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk5zm\" (UniqueName: \"kubernetes.io/projected/ec345887-2f41-4a39-ab33-6f3227339294-kube-api-access-hk5zm\") pod \"maas-api-revoke-sub-38658d1d-npl2p\" (UID: \"ec345887-2f41-4a39-ab33-6f3227339294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"
Oct 29 22:28:54.733853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.721646    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec345887-2f41-4a39-ab33-6f3227339294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38658d1d-npl2p\" (UID: \"ec345887-2f41-4a39-ab33-6f3227339294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"
Oct 29 22:28:54.733853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.721675    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec345887-2f41-4a39-ab33-6f3227339294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38658d1d-npl2p\" (UID: \"ec345887-2f41-4a39-ab33-6f3227339294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"
Oct 29 22:28:54.823229 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.823187    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk5zm\" (UniqueName: \"kubernetes.io/projected/ec345887-2f41-4a39-ab33-6f3227339294-kube-api-access-hk5zm\") pod \"maas-api-revoke-sub-38658d1d-npl2p\" (UID: \"ec345887-2f41-4a39-ab33-6f3227339294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"
Oct 29 22:28:54.823439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.823317    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec345887-2f41-4a39-ab33-6f3227339294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38658d1d-npl2p\" (UID: \"ec345887-2f41-4a39-ab33-6f3227339294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"
Oct 29 22:28:54.823439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.823349    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec345887-2f41-4a39-ab33-6f3227339294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38658d1d-npl2p\" (UID: \"ec345887-2f41-4a39-ab33-6f3227339294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"
Oct 29 22:28:54.824453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.824429    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec345887-2f41-4a39-ab33-6f3227339294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38658d1d-npl2p\" (UID: \"ec345887-2f41-4a39-ab33-6f3227339294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"
Oct 29 22:28:54.827210 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.827181    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec345887-2f41-4a39-ab33-6f3227339294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38658d1d-npl2p\" (UID: \"ec345887-2f41-4a39-ab33-6f3227339294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"
Oct 29 22:28:54.904413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.904213    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk5zm\" (UniqueName: \"kubernetes.io/projected/ec345887-2f41-4a39-ab33-6f3227339294-kube-api-access-hk5zm\") pod \"maas-api-revoke-sub-38658d1d-npl2p\" (UID: \"ec345887-2f41-4a39-ab33-6f3227339294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"
Oct 29 22:28:54.917365 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.917337    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"
Oct 29 22:28:54.942392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.941965    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:28:54.942392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:54.942022    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn" podUID="e3bc3b1a-f61c-4835-98e5-b9efccd4470e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 29 22:28:55.260606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:55.260515    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:28:55.278871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:55.271310    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw" podUID="20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 29 22:28:55.364355 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:55.364296    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"]
Oct 29 22:28:55.404578 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:28:55.404542    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec345887_2f41_4a39_ab33_6f3227339294.slice/crio-78621068b710bb33dd0e66cb3b072bc6345a6b1d8a835e21170f6e3f78f6b258 WatchSource:0}: Error finding container 78621068b710bb33dd0e66cb3b072bc6345a6b1d8a835e21170f6e3f78f6b258: Status 404 returned error can't find the container with id 78621068b710bb33dd0e66cb3b072bc6345a6b1d8a835e21170f6e3f78f6b258
Oct 29 22:28:55.959963 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:55.959921    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:28:55.960215 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:55.960186    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn" podUID="e3bc3b1a-f61c-4835-98e5-b9efccd4470e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 29 22:28:56.255904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:56.255293    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:28:56.255904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:56.255371    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw" podUID="20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 29 22:28:56.321624 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:56.321591    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p" event={"ID":"ec345887-2f41-4a39-ab33-6f3227339294","Type":"ContainerStarted","Data":"78621068b710bb33dd0e66cb3b072bc6345a6b1d8a835e21170f6e3f78f6b258"}
Oct 29 22:28:56.962838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:56.947413    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:28:56.962838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:56.947490    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn" podUID="e3bc3b1a-f61c-4835-98e5-b9efccd4470e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 29 22:28:57.249896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:57.249223    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:28:57.249896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:57.249299    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw" podUID="20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 29 22:28:57.942154 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:57.942113    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:28:57.942684 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:57.942200    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn" podUID="e3bc3b1a-f61c-4835-98e5-b9efccd4470e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 29 22:28:58.263326 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:58.263245    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:58.264237 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:58.264216    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:58.265376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:58.265356    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:28:58.946111 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:58.946072    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:58.946493 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:58.946390    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:58.947130 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:58.947112    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:28:59.344544 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:59.344504    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p" event={"ID":"ec345887-2f41-4a39-ab33-6f3227339294","Type":"ContainerStarted","Data":"9b3128d1a077379af21676ea4392fc63a15c2c92d9fea4486fa725d1475b00e0"}
Oct 29 22:28:59.387018 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:28:59.386957    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p" podStartSLOduration=1.9758106340000001 podStartE2EDuration="5.386941623s" totalImagesPullingTime="3.411130989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:59.384992812 +0000 UTC m=+878.296436379" watchObservedRunningTime="2026-10-29 22:28:59.386941623 +0000 UTC m=+878.298385155"
Oct 29 22:29:00.353329 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:00.353286    4559 generic.go:356] "Generic (PLEG): container finished" podID="ec345887-2f41-4a39-ab33-6f3227339294" containerID="9b3128d1a077379af21676ea4392fc63a15c2c92d9fea4486fa725d1475b00e0" exitCode=0
Oct 29 22:29:00.353869 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:00.353376    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p" event={"ID":"ec345887-2f41-4a39-ab33-6f3227339294","Type":"ContainerDied","Data":"9b3128d1a077379af21676ea4392fc63a15c2c92d9fea4486fa725d1475b00e0"}
Oct 29 22:29:01.513301 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:01.513275    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"
Oct 29 22:29:01.666478 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:01.666423    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec345887-2f41-4a39-ab33-6f3227339294-kube-api-access-hk5zm\" (UniqueName: \"kubernetes.io/projected/ec345887-2f41-4a39-ab33-6f3227339294-kube-api-access-hk5zm\") pod \"ec345887-2f41-4a39-ab33-6f3227339294\" (UID: \"ec345887-2f41-4a39-ab33-6f3227339294\") "
Oct 29 22:29:01.666689 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:01.666560    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec345887-2f41-4a39-ab33-6f3227339294-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec345887-2f41-4a39-ab33-6f3227339294-maas-api-openshift-ca\") pod \"ec345887-2f41-4a39-ab33-6f3227339294\" (UID: \"ec345887-2f41-4a39-ab33-6f3227339294\") "
Oct 29 22:29:01.666760 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:01.666744    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec345887-2f41-4a39-ab33-6f3227339294-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec345887-2f41-4a39-ab33-6f3227339294-maas-api-xks-ca\") pod \"ec345887-2f41-4a39-ab33-6f3227339294\" (UID: \"ec345887-2f41-4a39-ab33-6f3227339294\") "
Oct 29 22:29:01.667025 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:01.666992    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec345887-2f41-4a39-ab33-6f3227339294-maas-api-openshift-ca" pod "ec345887-2f41-4a39-ab33-6f3227339294" (UID: "ec345887-2f41-4a39-ab33-6f3227339294"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:29:01.667273 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:01.667255    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec345887-2f41-4a39-ab33-6f3227339294-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:29:01.669019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:01.668992    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec345887-2f41-4a39-ab33-6f3227339294-maas-api-xks-ca" pod "ec345887-2f41-4a39-ab33-6f3227339294" (UID: "ec345887-2f41-4a39-ab33-6f3227339294"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:29:01.669151 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:01.669070    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec345887-2f41-4a39-ab33-6f3227339294-kube-api-access-hk5zm" pod "ec345887-2f41-4a39-ab33-6f3227339294" (UID: "ec345887-2f41-4a39-ab33-6f3227339294"). InnerVolumeSpecName "kube-api-access-hk5zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:29:01.768648 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:01.768604    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec345887-2f41-4a39-ab33-6f3227339294-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:29:01.768648 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:01.768639    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk5zm\" (UniqueName: \"kubernetes.io/projected/ec345887-2f41-4a39-ab33-6f3227339294-kube-api-access-hk5zm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:29:02.366999 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:02.366962    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p" event={"ID":"ec345887-2f41-4a39-ab33-6f3227339294","Type":"ContainerDied","Data":"78621068b710bb33dd0e66cb3b072bc6345a6b1d8a835e21170f6e3f78f6b258"}
Oct 29 22:29:02.367196 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:02.367007    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"
Oct 29 22:29:02.367196 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:02.367018    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78621068b710bb33dd0e66cb3b072bc6345a6b1d8a835e21170f6e3f78f6b258"
Oct 29 22:29:03.890900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:03.890863    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"]
Oct 29 22:29:03.891666 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:03.891644    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ec345887-2f41-4a39-ab33-6f3227339294" containerName="revoke-keys"
Oct 29 22:29:03.891666 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:03.891665    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec345887-2f41-4a39-ab33-6f3227339294" containerName="revoke-keys"
Oct 29 22:29:03.891867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:03.891845    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec345887-2f41-4a39-ab33-6f3227339294" containerName="revoke-keys"
Oct 29 22:29:03.895862 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:03.895837    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"
Oct 29 22:29:03.916955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:03.916929    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:29:03.929747 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:03.929710    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:29:03.932354 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:03.932329    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"]
Oct 29 22:29:03.995035 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:03.994999    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71962013-6l9m4\" (UID: \"806d27e8-e0bd-4f8a-ab8e-067352d7f75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"
Oct 29 22:29:03.995217 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:03.995120    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x9jg\" (UniqueName: \"kubernetes.io/projected/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-kube-api-access-5x9jg\") pod \"maas-api-revoke-sub-71962013-6l9m4\" (UID: \"806d27e8-e0bd-4f8a-ab8e-067352d7f75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"
Oct 29 22:29:03.995217 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:03.995199    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71962013-6l9m4\" (UID: \"806d27e8-e0bd-4f8a-ab8e-067352d7f75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"
Oct 29 22:29:04.096620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:04.096578    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71962013-6l9m4\" (UID: \"806d27e8-e0bd-4f8a-ab8e-067352d7f75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"
Oct 29 22:29:04.096778 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:04.096634    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x9jg\" (UniqueName: \"kubernetes.io/projected/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-kube-api-access-5x9jg\") pod \"maas-api-revoke-sub-71962013-6l9m4\" (UID: \"806d27e8-e0bd-4f8a-ab8e-067352d7f75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"
Oct 29 22:29:04.096778 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:04.096674    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71962013-6l9m4\" (UID: \"806d27e8-e0bd-4f8a-ab8e-067352d7f75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"
Oct 29 22:29:04.097476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:04.097456    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71962013-6l9m4\" (UID: \"806d27e8-e0bd-4f8a-ab8e-067352d7f75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"
Oct 29 22:29:04.099267 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:04.099247    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71962013-6l9m4\" (UID: \"806d27e8-e0bd-4f8a-ab8e-067352d7f75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"
Oct 29 22:29:04.117768 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:04.117742    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x9jg\" (UniqueName: \"kubernetes.io/projected/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-kube-api-access-5x9jg\") pod \"maas-api-revoke-sub-71962013-6l9m4\" (UID: \"806d27e8-e0bd-4f8a-ab8e-067352d7f75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"
Oct 29 22:29:04.215397 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:04.215303    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"
Oct 29 22:29:04.520033 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:04.520007    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"]
Oct 29 22:29:04.522120 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:29:04.522089    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806d27e8_e0bd_4f8a_ab8e_067352d7f75f.slice/crio-d4e7be0c3fec1873155788d3d0006bddfea783b3c826ae7dfeb098574095ccd3 WatchSource:0}: Error finding container d4e7be0c3fec1873155788d3d0006bddfea783b3c826ae7dfeb098574095ccd3: Status 404 returned error can't find the container with id d4e7be0c3fec1873155788d3d0006bddfea783b3c826ae7dfeb098574095ccd3
Oct 29 22:29:05.417500 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:05.417467    4559 generic.go:356] "Generic (PLEG): container finished" podID="806d27e8-e0bd-4f8a-ab8e-067352d7f75f" containerID="15467d342b814c7fc3bd5c1a01577d17983e43b5fbdb34a76d7e5bbdd12d6d4f" exitCode=0
Oct 29 22:29:05.417922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:05.417556    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4" event={"ID":"806d27e8-e0bd-4f8a-ab8e-067352d7f75f","Type":"ContainerDied","Data":"15467d342b814c7fc3bd5c1a01577d17983e43b5fbdb34a76d7e5bbdd12d6d4f"}
Oct 29 22:29:05.417922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:05.417626    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4" event={"ID":"806d27e8-e0bd-4f8a-ab8e-067352d7f75f","Type":"ContainerStarted","Data":"d4e7be0c3fec1873155788d3d0006bddfea783b3c826ae7dfeb098574095ccd3"}
Oct 29 22:29:06.576206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:06.576177    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"
Oct 29 22:29:06.628896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:06.628868    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-maas-api-xks-ca\") pod \"806d27e8-e0bd-4f8a-ab8e-067352d7f75f\" (UID: \"806d27e8-e0bd-4f8a-ab8e-067352d7f75f\") "
Oct 29 22:29:06.629093 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:06.628961    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-maas-api-openshift-ca\") pod \"806d27e8-e0bd-4f8a-ab8e-067352d7f75f\" (UID: \"806d27e8-e0bd-4f8a-ab8e-067352d7f75f\") "
Oct 29 22:29:06.629093 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:06.629077    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-kube-api-access-5x9jg\" (UniqueName: \"kubernetes.io/projected/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-kube-api-access-5x9jg\") pod \"806d27e8-e0bd-4f8a-ab8e-067352d7f75f\" (UID: \"806d27e8-e0bd-4f8a-ab8e-067352d7f75f\") "
Oct 29 22:29:06.629393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:06.629366    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-maas-api-openshift-ca" pod "806d27e8-e0bd-4f8a-ab8e-067352d7f75f" (UID: "806d27e8-e0bd-4f8a-ab8e-067352d7f75f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:29:06.629822 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:06.629789    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:29:06.631827 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:06.631790    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-kube-api-access-5x9jg" pod "806d27e8-e0bd-4f8a-ab8e-067352d7f75f" (UID: "806d27e8-e0bd-4f8a-ab8e-067352d7f75f"). InnerVolumeSpecName "kube-api-access-5x9jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:29:06.631913 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:06.631798    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-maas-api-xks-ca" pod "806d27e8-e0bd-4f8a-ab8e-067352d7f75f" (UID: "806d27e8-e0bd-4f8a-ab8e-067352d7f75f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:29:06.731135 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:06.731057    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:29:06.731135 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:06.731085    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x9jg\" (UniqueName: \"kubernetes.io/projected/806d27e8-e0bd-4f8a-ab8e-067352d7f75f-kube-api-access-5x9jg\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:29:07.433059 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:07.433022    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"
Oct 29 22:29:07.433262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:07.433019    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4" event={"ID":"806d27e8-e0bd-4f8a-ab8e-067352d7f75f","Type":"ContainerDied","Data":"d4e7be0c3fec1873155788d3d0006bddfea783b3c826ae7dfeb098574095ccd3"}
Oct 29 22:29:07.433262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:07.433163    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4e7be0c3fec1873155788d3d0006bddfea783b3c826ae7dfeb098574095ccd3"
Oct 29 22:29:13.694373 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:29:13.694335    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:29:19.429543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:19.429509    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:29:21.354042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:21.354009    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:29:21.354560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:21.354209    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:29:35.081281 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.081242    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"]
Oct 29 22:29:35.082746 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.082688    4559 cpu_manager.go:414] "containerMap: removing container" podUID="806d27e8-e0bd-4f8a-ab8e-067352d7f75f" containerName="revoke-keys"
Oct 29 22:29:35.082975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.082960    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="806d27e8-e0bd-4f8a-ab8e-067352d7f75f" containerName="revoke-keys"
Oct 29 22:29:35.083315 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.083285    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="806d27e8-e0bd-4f8a-ab8e-067352d7f75f" containerName="revoke-keys"
Oct 29 22:29:35.114835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.108007    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.123171 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.123143    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"]
Oct 29 22:29:35.179738 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.179702    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.179968 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.179750    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.179968 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.179803    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.180091 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.180068    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpbx8\" (UniqueName: \"kubernetes.io/projected/7f3dffab-440d-4fb7-81bb-9c63ed54d908-kube-api-access-wpbx8\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.180148 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.180119    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.180213 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.180191    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.180279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.180255    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.180352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.180336    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.180391 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.180366    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.281545 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.281502    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.281746 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.281566    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.281746 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.281647    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.281910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.281784    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbx8\" (UniqueName: \"kubernetes.io/projected/7f3dffab-440d-4fb7-81bb-9c63ed54d908-kube-api-access-wpbx8\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.281910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.281840    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.281910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.281884    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.282068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.281910    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.282068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.281929    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.282068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.281944    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.282068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.281987    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.282378 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.282354    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.282554 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.282465    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.282743 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.282724    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.283023 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.282794    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.284484 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.284461    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.284594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.284463    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.301081 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.301051    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.301268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.301243    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpbx8\" (UniqueName: \"kubernetes.io/projected/7f3dffab-440d-4fb7-81bb-9c63ed54d908-kube-api-access-wpbx8\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.354126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.354034    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"]
Oct 29 22:29:35.360437 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.360413    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.383353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.383077    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.383353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.383148    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5a7c39-6931-4960-932c-910ad8d3c53a-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.383353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.383194    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72l99\" (UniqueName: \"kubernetes.io/projected/ca5a7c39-6931-4960-932c-910ad8d3c53a-kube-api-access-72l99\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.383641 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.383446    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"]
Oct 29 22:29:35.383733 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.383709    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.383800 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.383748    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.383879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.383841    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.383879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.383872    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.383983 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.383943    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.384035 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.384001    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.433731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.433695    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:35.473579 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.473546    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"]
Oct 29 22:29:35.478842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.478003    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.484782 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.484758    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.485060 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.485042    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.485184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.485174    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.485258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.485237    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.485314 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.485305    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5a7c39-6931-4960-932c-910ad8d3c53a-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.485404 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.485388    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72l99\" (UniqueName: \"kubernetes.io/projected/ca5a7c39-6931-4960-932c-910ad8d3c53a-kube-api-access-72l99\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.485530 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.485519    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.485602 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.485592    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.485697 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.485688    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.485759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.485751    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.485882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.485414    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.486165 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.486142    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5a7c39-6931-4960-932c-910ad8d3c53a-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.486245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.486226    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.486313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.486272    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.489068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.488614    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.489494 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.489476    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.500554 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.500521    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"]
Oct 29 22:29:35.511653 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.511629    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.513456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.513437    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72l99\" (UniqueName: \"kubernetes.io/projected/ca5a7c39-6931-4960-932c-910ad8d3c53a-kube-api-access-72l99\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.586705 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.586669    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kmxp\" (UniqueName: \"kubernetes.io/projected/8fad0702-72b6-4352-a752-f9aa98638048-kube-api-access-4kmxp\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.586941 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.586822    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.587033 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.586949    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.587033 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.587020    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.587151 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.587113    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.587151 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.587143    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fad0702-72b6-4352-a752-f9aa98638048-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.587248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.587221    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fad0702-72b6-4352-a752-f9aa98638048-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.587310 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.587297    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.587376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.587357    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fad0702-72b6-4352-a752-f9aa98638048-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.678557 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.678525    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:35.688331 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.688299    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.688470 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.688338    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fad0702-72b6-4352-a752-f9aa98638048-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.688470 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.688385    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fad0702-72b6-4352-a752-f9aa98638048-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.688470 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.688463    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.688621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.688497    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fad0702-72b6-4352-a752-f9aa98638048-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.688621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.688534    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kmxp\" (UniqueName: \"kubernetes.io/projected/8fad0702-72b6-4352-a752-f9aa98638048-kube-api-access-4kmxp\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.688727 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.688632    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.688727 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.688676    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.688872 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.688736    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.688944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.688875    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.689113 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.689093    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.689175 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.689124    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.689175 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.689144    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.689434 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.689413    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fad0702-72b6-4352-a752-f9aa98638048-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.691327 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.691221    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fad0702-72b6-4352-a752-f9aa98638048-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.691386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.691340    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.708867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.708837    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fad0702-72b6-4352-a752-f9aa98638048-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.709036 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.708890    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kmxp\" (UniqueName: \"kubernetes.io/projected/8fad0702-72b6-4352-a752-f9aa98638048-kube-api-access-4kmxp\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.766207 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.766161    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"]
Oct 29 22:29:35.771015 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:29:35.770966    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3dffab_440d_4fb7_81bb_9c63ed54d908.slice/crio-a9eacf5668cdc107a0df44071aae5d77d419955d4be506fed25ca1b306693e14 WatchSource:0}: Error finding container a9eacf5668cdc107a0df44071aae5d77d419955d4be506fed25ca1b306693e14: Status 404 returned error can't find the container with id a9eacf5668cdc107a0df44071aae5d77d419955d4be506fed25ca1b306693e14
Oct 29 22:29:35.774038 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.773996    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:29:35.774203 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.774182    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:29:35.774295 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.774277    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:29:35.811006 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.810974    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:35.842714 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.842681    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"]
Oct 29 22:29:35.847972 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.847946    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.887449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.887418    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"]
Oct 29 22:29:35.901172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.890860    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.901172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.890960    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.901172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.891037    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd2kv\" (UniqueName: \"kubernetes.io/projected/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-kube-api-access-dd2kv\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.901172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.891107    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.901172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.891130    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.901172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.891178    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.901172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.891224    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.901172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.891260    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.901172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.891291    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.996245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.992640    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.996245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.992718    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd2kv\" (UniqueName: \"kubernetes.io/projected/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-kube-api-access-dd2kv\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.996245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.992765    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.996245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.992789    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.996245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.992856    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.996245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.992886    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.996245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.992920    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.996245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.992944    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.996245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.992974    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.996245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.993395    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.996245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.994151    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.996245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.994543    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.996245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.994966    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.996245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.995236    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:35.997151 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.997141    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:36.000331 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:35.999306    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:36.005165 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.004292    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"]
Oct 29 22:29:36.018235 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.014351    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:29:36.018235 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.014430    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:29:36.018235 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.014468    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:29:36.019732 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.019528    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:36.019732 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.019683    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd2kv\" (UniqueName: \"kubernetes.io/projected/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-kube-api-access-dd2kv\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:36.051954 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.051925    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:29:36.186146 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.186107    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:36.211399 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.210878    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:29:36.211399 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.210971    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:29:36.211399 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.211009    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:29:36.213054 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.213034    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"]
Oct 29 22:29:36.576533 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.576486    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"]
Oct 29 22:29:36.592896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.592436    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:29:36.592896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.592505    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:29:36.592896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.592534    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:29:36.640667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.640619    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks" event={"ID":"7f3dffab-440d-4fb7-81bb-9c63ed54d908","Type":"ContainerStarted","Data":"ffc4eb0a936fcbd2f7d4753d6a50390117a552d7cf382444b7ad6bab7125ced2"}
Oct 29 22:29:36.640882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.640698    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks" event={"ID":"7f3dffab-440d-4fb7-81bb-9c63ed54d908","Type":"ContainerStarted","Data":"a9eacf5668cdc107a0df44071aae5d77d419955d4be506fed25ca1b306693e14"}
Oct 29 22:29:36.654836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.650987    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2" event={"ID":"8fad0702-72b6-4352-a752-f9aa98638048","Type":"ContainerStarted","Data":"111e1c2cce2fff8039e0e6d70f4d7adc552cf52942dba14a05720e06c5516a83"}
Oct 29 22:29:36.654836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.651053    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2" event={"ID":"8fad0702-72b6-4352-a752-f9aa98638048","Type":"ContainerStarted","Data":"ce4dd3157a2428547fa7296452e77c91ceee3784c03217090fcc79084c642767"}
Oct 29 22:29:36.658834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.658523    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82" event={"ID":"6a3bfdf3-5fca-44b9-88d7-bf525139ee93","Type":"ContainerStarted","Data":"c7c34923d15368d754f5bfa8c08a60505348a698d55a31275d74282c27b225d5"}
Oct 29 22:29:36.665836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.660980    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8" event={"ID":"ca5a7c39-6931-4960-932c-910ad8d3c53a","Type":"ContainerStarted","Data":"5575c0b20a085faef27efaac24aa1d614172f76c698dd1f021cfa78f44b8db78"}
Oct 29 22:29:36.665836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.661036    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8" event={"ID":"ca5a7c39-6931-4960-932c-910ad8d3c53a","Type":"ContainerStarted","Data":"9e9f369a7c00f6f0e1a8e0f40f514b54680ec2eebb808a66a7104e85fd54437b"}
Oct 29 22:29:36.683938 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.679956    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:36.705927 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.701477    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks" podStartSLOduration=1.70145906 podStartE2EDuration="1.70145906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:36.699451474 +0000 UTC m=+915.610895030" watchObservedRunningTime="2026-10-29 22:29:36.70145906 +0000 UTC m=+915.612902614"
Oct 29 22:29:36.811896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.795569    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2" podStartSLOduration=1.795544679 podStartE2EDuration="1.795544679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:36.745734431 +0000 UTC m=+915.657177986" watchObservedRunningTime="2026-10-29 22:29:36.795544679 +0000 UTC m=+915.706988238"
Oct 29 22:29:36.829837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.811310    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:36.830442 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:36.830357    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8" podStartSLOduration=1.830332251 podStartE2EDuration="1.830332251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:36.824183566 +0000 UTC m=+915.735627102" watchObservedRunningTime="2026-10-29 22:29:36.830332251 +0000 UTC m=+915.741775807"
Oct 29 22:29:37.435842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:37.435789    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:37.436518 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:37.436436    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 29 22:29:37.436695 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:37.436671    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks" podUID="7f3dffab-440d-4fb7-81bb-9c63ed54d908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 29 22:29:37.728592 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:37.726225    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82" event={"ID":"6a3bfdf3-5fca-44b9-88d7-bf525139ee93","Type":"ContainerStarted","Data":"1c153840f8535885d6955996c97c2fc506a2450fff700e6c8ed7ecf1be7fc0e8"}
Oct 29 22:29:37.729010 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:37.728981    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 29 22:29:37.729212 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:37.729189    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8" podUID="ca5a7c39-6931-4960-932c-910ad8d3c53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 29 22:29:37.822580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:37.822509    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82" podStartSLOduration=2.822486778 podStartE2EDuration="2.822486778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:37.789270247 +0000 UTC m=+916.700713803" watchObservedRunningTime="2026-10-29 22:29:37.822486778 +0000 UTC m=+916.733930333"
Oct 29 22:29:37.833008 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:37.830959    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:29:37.833290 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:37.833260    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2" podUID="8fad0702-72b6-4352-a752-f9aa98638048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 22:29:38.188841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:38.188480    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:38.188841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:38.188610    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:29:38.188841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:38.188653    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82" podUID="6a3bfdf3-5fca-44b9-88d7-bf525139ee93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 29 22:29:38.434382 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:38.434347    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 29 22:29:38.434673 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:38.434644    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks" podUID="7f3dffab-440d-4fb7-81bb-9c63ed54d908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 29 22:29:38.682328 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:38.682286    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 29 22:29:38.682993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:38.682961    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8" podUID="ca5a7c39-6931-4960-932c-910ad8d3c53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 29 22:29:38.823068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:38.823032    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:29:38.823392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:38.823326    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2" podUID="8fad0702-72b6-4352-a752-f9aa98638048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 22:29:39.188251 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:39.188212    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:29:39.189297 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:39.189261    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82" podUID="6a3bfdf3-5fca-44b9-88d7-bf525139ee93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 29 22:29:39.481891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:39.455752    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 29 22:29:39.481891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:39.455842    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks" podUID="7f3dffab-440d-4fb7-81bb-9c63ed54d908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 29 22:29:39.633296 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:39.633247    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:39.657554 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:39.657511    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:39.715837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:39.690722    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 29 22:29:39.716456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:39.716414    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8" podUID="ca5a7c39-6931-4960-932c-910ad8d3c53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 29 22:29:39.812013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:39.811902    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:39.813155 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:39.813120    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:29:39.813269 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:39.813176    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2" podUID="8fad0702-72b6-4352-a752-f9aa98638048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 22:29:39.847116 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:39.847070    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:29:39.990988 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:39.990938    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:40.198988 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:40.198947    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:29:40.199400 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:40.199368    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82" podUID="6a3bfdf3-5fca-44b9-88d7-bf525139ee93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 29 22:29:40.440835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:40.440246    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 29 22:29:40.440835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:40.440320    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks" podUID="7f3dffab-440d-4fb7-81bb-9c63ed54d908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 29 22:29:40.670388 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:40.668256    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:29:40.670388 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:40.668801    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:29:40.686841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:40.686299    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 29 22:29:40.686841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:40.686364    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8" podUID="ca5a7c39-6931-4960-932c-910ad8d3c53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 29 22:29:40.813911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:40.813266    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:29:40.813911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:40.813349    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2" podUID="8fad0702-72b6-4352-a752-f9aa98638048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 22:29:40.873009 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:40.872963    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:29:40.873685 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:40.873661    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:29:41.195002 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:41.189646    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:29:41.195002 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:41.189693    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82" podUID="6a3bfdf3-5fca-44b9-88d7-bf525139ee93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 29 22:29:41.503905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:41.499420    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:29:41.523647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:41.523612    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 29 22:29:41.523944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:41.523914    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks" podUID="7f3dffab-440d-4fb7-81bb-9c63ed54d908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 29 22:29:41.669132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:41.669083    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:41.682221 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:41.682184    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 29 22:29:41.691549 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:41.691496    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8" podUID="ca5a7c39-6931-4960-932c-910ad8d3c53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 29 22:29:41.766492 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:41.766304    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:41.767001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:41.766355    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:41.917768 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:41.917737    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:29:41.918382 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:41.918351    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2" podUID="8fad0702-72b6-4352-a752-f9aa98638048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 22:29:41.992630 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:41.992576    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 22:29:42.193834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:42.193245    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:29:42.193834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:42.193328    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82" podUID="6a3bfdf3-5fca-44b9-88d7-bf525139ee93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 29 22:29:42.439116 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:42.438979    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 29 22:29:42.439905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:42.439397    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks" podUID="7f3dffab-440d-4fb7-81bb-9c63ed54d908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 29 22:29:42.511586 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:42.511463    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:29:42.697837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:42.684314    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 29 22:29:42.698720 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:42.698670    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8" podUID="ca5a7c39-6931-4960-932c-910ad8d3c53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 29 22:29:42.826894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:42.826045    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:29:42.838110 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:42.838068    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:29:42.860755 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:42.860721    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:29:42.861075 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:42.861038    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2" podUID="8fad0702-72b6-4352-a752-f9aa98638048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 22:29:43.020035 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:43.020002    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:29:43.233837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:43.229974    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:29:43.233837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:43.230036    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82" podUID="6a3bfdf3-5fca-44b9-88d7-bf525139ee93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 29 22:29:43.482864 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:43.461912    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 29 22:29:43.482864 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:43.461975    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks" podUID="7f3dffab-440d-4fb7-81bb-9c63ed54d908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 29 22:29:43.698117 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:43.698078    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 29 22:29:43.727730 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:43.711345    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8" podUID="ca5a7c39-6931-4960-932c-910ad8d3c53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 29 22:29:43.821838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:43.820876    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:29:43.821838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:43.820936    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2" podUID="8fad0702-72b6-4352-a752-f9aa98638048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 22:29:43.918839 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:29:43.918243    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:29:44.193679 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:44.193625    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:29:44.196853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:44.194877    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82" podUID="6a3bfdf3-5fca-44b9-88d7-bf525139ee93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 29 22:29:44.504954 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:44.459923    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 29 22:29:44.504954 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:44.460054    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks" podUID="7f3dffab-440d-4fb7-81bb-9c63ed54d908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 29 22:29:44.718988 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:44.718944    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 29 22:29:44.719239 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:44.719216    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8" podUID="ca5a7c39-6931-4960-932c-910ad8d3c53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 29 22:29:44.812764 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:44.812250    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:29:44.812764 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:44.812317    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2" podUID="8fad0702-72b6-4352-a752-f9aa98638048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 22:29:45.187839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:45.187479    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:29:45.187839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:45.187562    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82" podUID="6a3bfdf3-5fca-44b9-88d7-bf525139ee93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 29 22:29:45.444546 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:45.444459    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:45.445409 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:45.445389    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:45.446434 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:45.446409    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:29:45.686293 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:45.686262    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:45.687741 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:45.687719    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:45.688458 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:45.688431    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:29:45.912413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:45.912378    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:45.913280 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:45.913259    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:45.914337 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:45.914313    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:29:46.409841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:46.409720    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:46.411337 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:46.411307    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:46.413543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:46.413520    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:29:50.587111 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.587066    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"]
Oct 29 22:29:50.591609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.591589    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.620752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.620719    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.621002 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.620786    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.621002 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.620871    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.621002 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.620914    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.621002 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.620983    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztjkh\" (UniqueName: \"kubernetes.io/projected/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-kube-api-access-ztjkh\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.621240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.621055    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.621240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.621209    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.621352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.621302    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.621352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.621326    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.624959 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.624930    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"]
Oct 29 22:29:50.722654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.722613    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.722891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.722742    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.722891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.722767    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.722891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.722802    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.723082 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.722912    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.723082 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.722947    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.723082 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.722987    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.723082 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.723014    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztjkh\" (UniqueName: \"kubernetes.io/projected/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-kube-api-access-ztjkh\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.723082 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.723043    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.723352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.723234    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.723352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.723258    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.723352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.723298    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.723504 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.723401    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.724202 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.724178    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.725426 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.725401    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.725844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.725801    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.742927 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.742885    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.743073 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.743032    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztjkh\" (UniqueName: \"kubernetes.io/projected/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-kube-api-access-ztjkh\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:50.906654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:50.906621    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:51.231301 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:51.231277    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"]
Oct 29 22:29:51.232094 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:29:51.232074    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c5c47b_9b0c_4d07_bfca_5eacb7d77f0d.slice/crio-202b01fa078656c6e4d90b0c3e962479af81969ae7850af77ccf0fdc433ec283 WatchSource:0}: Error finding container 202b01fa078656c6e4d90b0c3e962479af81969ae7850af77ccf0fdc433ec283: Status 404 returned error can't find the container with id 202b01fa078656c6e4d90b0c3e962479af81969ae7850af77ccf0fdc433ec283
Oct 29 22:29:51.234624 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:51.234593    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:29:51.234729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:51.234675    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:29:51.234787 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:51.234725    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:29:51.629240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:51.629151    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl" event={"ID":"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d","Type":"ContainerStarted","Data":"4b7700b08a80e8b4595d7849c3dc29024d3ab55e8569b35715e6a0d3891bfce8"}
Oct 29 22:29:51.629240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:51.629215    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl" event={"ID":"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d","Type":"ContainerStarted","Data":"202b01fa078656c6e4d90b0c3e962479af81969ae7850af77ccf0fdc433ec283"}
Oct 29 22:29:51.703023 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:51.702966    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl" podStartSLOduration=1.702948062 podStartE2EDuration="1.702948062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:51.685343241 +0000 UTC m=+930.596786795" watchObservedRunningTime="2026-10-29 22:29:51.702948062 +0000 UTC m=+930.614391615"
Oct 29 22:29:51.906783 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:51.906733    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:52.911024 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:52.910973    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 29 22:29:52.911754 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:52.911714    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl" podUID="d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 29 22:29:53.907288 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:53.907249    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 29 22:29:53.907492 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:53.907320    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl" podUID="d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 29 22:29:54.907606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:54.907569    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 29 22:29:54.908148 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:54.907638    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl" podUID="d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 29 22:29:55.907548 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:55.907499    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 29 22:29:55.907744 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:55.907576    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl" podUID="d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 29 22:29:56.907538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:56.907491    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 29 22:29:56.907702 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:56.907575    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl" podUID="d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 29 22:29:58.360074 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:58.360036    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:58.361115 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:58.361088    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:58.362424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:58.362402    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 22:29:58.711121 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:29:58.711087    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:30:00.154546 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.154496    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw"]
Oct 29 22:30:00.158790 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.158765    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw"
Oct 29 22:30:00.175223 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.175193    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b"]
Oct 29 22:30:00.193152 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.193119    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:30:00.200579 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.200538    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687"]
Oct 29 22:30:00.200843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.200824    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b"
Oct 29 22:30:00.220623 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.220450    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw"]
Oct 29 22:30:00.220623 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.220502    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b"]
Oct 29 22:30:00.220848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.220628    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687"
Oct 29 22:30:00.233974 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.233944    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687"]
Oct 29 22:30:00.315948 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.315911    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4edf698a-ccad-4baf-a7dd-0e91e2f2409a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b\" (UID: \"4edf698a-ccad-4baf-a7dd-0e91e2f2409a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b"
Oct 29 22:30:00.316163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.316062    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rxc6\" (UniqueName: \"kubernetes.io/projected/a29150a5-5b05-493e-9968-cf13fd43b2d6-kube-api-access-7rxc6\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687\" (UID: \"a29150a5-5b05-493e-9968-cf13fd43b2d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687"
Oct 29 22:30:00.316163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.316110    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888550-s2xhw\" (UID: \"24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw"
Oct 29 22:30:00.316291 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.316169    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvnb4\" (UniqueName: \"kubernetes.io/projected/4edf698a-ccad-4baf-a7dd-0e91e2f2409a-kube-api-access-xvnb4\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b\" (UID: \"4edf698a-ccad-4baf-a7dd-0e91e2f2409a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b"
Oct 29 22:30:00.316291 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.316234    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29150a5-5b05-493e-9968-cf13fd43b2d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687\" (UID: \"a29150a5-5b05-493e-9968-cf13fd43b2d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687"
Oct 29 22:30:00.316376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.316313    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9vhq\" (UniqueName: \"kubernetes.io/projected/24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc-kube-api-access-x9vhq\") pod \"maas-api-key-cleanup-29888550-s2xhw\" (UID: \"24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw"
Oct 29 22:30:00.417290 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.417199    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4edf698a-ccad-4baf-a7dd-0e91e2f2409a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b\" (UID: \"4edf698a-ccad-4baf-a7dd-0e91e2f2409a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b"
Oct 29 22:30:00.417472 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.417307    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rxc6\" (UniqueName: \"kubernetes.io/projected/a29150a5-5b05-493e-9968-cf13fd43b2d6-kube-api-access-7rxc6\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687\" (UID: \"a29150a5-5b05-493e-9968-cf13fd43b2d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687"
Oct 29 22:30:00.417472 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.417344    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888550-s2xhw\" (UID: \"24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw"
Oct 29 22:30:00.418094 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.418069    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4edf698a-ccad-4baf-a7dd-0e91e2f2409a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b\" (UID: \"4edf698a-ccad-4baf-a7dd-0e91e2f2409a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b"
Oct 29 22:30:00.418256 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.418069    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888550-s2xhw\" (UID: \"24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw"
Oct 29 22:30:00.418256 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.418187    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvnb4\" (UniqueName: \"kubernetes.io/projected/4edf698a-ccad-4baf-a7dd-0e91e2f2409a-kube-api-access-xvnb4\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b\" (UID: \"4edf698a-ccad-4baf-a7dd-0e91e2f2409a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b"
Oct 29 22:30:00.418256 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.418218    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29150a5-5b05-493e-9968-cf13fd43b2d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687\" (UID: \"a29150a5-5b05-493e-9968-cf13fd43b2d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687"
Oct 29 22:30:00.418370 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.418291    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vhq\" (UniqueName: \"kubernetes.io/projected/24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc-kube-api-access-x9vhq\") pod \"maas-api-key-cleanup-29888550-s2xhw\" (UID: \"24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw"
Oct 29 22:30:00.419001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.418982    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29150a5-5b05-493e-9968-cf13fd43b2d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687\" (UID: \"a29150a5-5b05-493e-9968-cf13fd43b2d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687"
Oct 29 22:30:00.468147 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.458598    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rxc6\" (UniqueName: \"kubernetes.io/projected/a29150a5-5b05-493e-9968-cf13fd43b2d6-kube-api-access-7rxc6\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687\" (UID: \"a29150a5-5b05-493e-9968-cf13fd43b2d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687"
Oct 29 22:30:00.491479 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.483510    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9vhq\" (UniqueName: \"kubernetes.io/projected/24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc-kube-api-access-x9vhq\") pod \"maas-api-key-cleanup-29888550-s2xhw\" (UID: \"24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw"
Oct 29 22:30:00.491479 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.483750    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw"
Oct 29 22:30:00.492590 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.492316    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvnb4\" (UniqueName: \"kubernetes.io/projected/4edf698a-ccad-4baf-a7dd-0e91e2f2409a-kube-api-access-xvnb4\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b\" (UID: \"4edf698a-ccad-4baf-a7dd-0e91e2f2409a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b"
Oct 29 22:30:00.521086 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.521050    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b"
Oct 29 22:30:00.555369 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.555124    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687"
Oct 29 22:30:00.894798 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:00.893394    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw"]
Oct 29 22:30:01.016364 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:01.016332    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b"]
Oct 29 22:30:01.027790 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:01.027718    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687"]
Oct 29 22:30:01.830537 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:01.830433    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b" event={"ID":"4edf698a-ccad-4baf-a7dd-0e91e2f2409a","Type":"ContainerStarted","Data":"a915a55281823b5d1c573f431e98b02079eb16bb4c73fd38a7cf0658701e9097"}
Oct 29 22:30:01.830537 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:01.830498    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b" event={"ID":"4edf698a-ccad-4baf-a7dd-0e91e2f2409a","Type":"ContainerStarted","Data":"8a27f549707d0fa60eeb35c4fc932d9e7c83880e8510def647dd167742ad65ad"}
Oct 29 22:30:01.832121 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:01.832094    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw" event={"ID":"24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc","Type":"ContainerStarted","Data":"6783f7cdcbe128186eda6e074b4612c39716d915a784f39c992eb48b2e66d093"}
Oct 29 22:30:01.832237 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:01.832138    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw" event={"ID":"24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc","Type":"ContainerStarted","Data":"aa7c9a940f4fa5259968184e975db06b5c44d5279faf7d338f2f9209bd5ec04f"}
Oct 29 22:30:01.836045 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:01.833746    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687" event={"ID":"a29150a5-5b05-493e-9968-cf13fd43b2d6","Type":"ContainerStarted","Data":"0b14344309e0d37e341c0c70f0cd851396b1227d60be1952cb510d19d0a08beb"}
Oct 29 22:30:01.836045 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:01.833794    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687" event={"ID":"a29150a5-5b05-493e-9968-cf13fd43b2d6","Type":"ContainerStarted","Data":"12393468b4a531bcbd1bd2307a1c8776a5a99b5807eaa5f6287e3156a8d0fde7"}
Oct 29 22:30:01.872475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:01.872419    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b" podStartSLOduration=1.872401864 podStartE2EDuration="1.872401864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:01.870374656 +0000 UTC m=+940.781818211" watchObservedRunningTime="2026-10-29 22:30:01.872401864 +0000 UTC m=+940.783845440"
Oct 29 22:30:01.911512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:01.911443    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw" podStartSLOduration=1.911421665 podStartE2EDuration="1.911421665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:01.90995839 +0000 UTC m=+940.821401943" watchObservedRunningTime="2026-10-29 22:30:01.911421665 +0000 UTC m=+940.822865222"
Oct 29 22:30:01.952297 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:01.952198    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687" podStartSLOduration=1.952180139 podStartE2EDuration="1.952180139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:01.949087434 +0000 UTC m=+940.860530988" watchObservedRunningTime="2026-10-29 22:30:01.952180139 +0000 UTC m=+940.863623693"
Oct 29 22:30:02.842647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:02.842607    4559 generic.go:356] "Generic (PLEG): container finished" podID="a29150a5-5b05-493e-9968-cf13fd43b2d6" containerID="0b14344309e0d37e341c0c70f0cd851396b1227d60be1952cb510d19d0a08beb" exitCode=60
Oct 29 22:30:02.843160 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:02.842701    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687" event={"ID":"a29150a5-5b05-493e-9968-cf13fd43b2d6","Type":"ContainerDied","Data":"0b14344309e0d37e341c0c70f0cd851396b1227d60be1952cb510d19d0a08beb"}
Oct 29 22:30:02.843160 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:02.843062    4559 scope.go:122] "RemoveContainer" containerID="0b14344309e0d37e341c0c70f0cd851396b1227d60be1952cb510d19d0a08beb"
Oct 29 22:30:02.844667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:02.844642    4559 generic.go:356] "Generic (PLEG): container finished" podID="4edf698a-ccad-4baf-a7dd-0e91e2f2409a" containerID="a915a55281823b5d1c573f431e98b02079eb16bb4c73fd38a7cf0658701e9097" exitCode=60
Oct 29 22:30:02.844843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:02.844724    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b" event={"ID":"4edf698a-ccad-4baf-a7dd-0e91e2f2409a","Type":"ContainerDied","Data":"a915a55281823b5d1c573f431e98b02079eb16bb4c73fd38a7cf0658701e9097"}
Oct 29 22:30:02.844984 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:02.844971    4559 scope.go:122] "RemoveContainer" containerID="a915a55281823b5d1c573f431e98b02079eb16bb4c73fd38a7cf0658701e9097"
Oct 29 22:30:02.850338 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:02.846420    4559 generic.go:356] "Generic (PLEG): container finished" podID="24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc" containerID="6783f7cdcbe128186eda6e074b4612c39716d915a784f39c992eb48b2e66d093" exitCode=60
Oct 29 22:30:02.850338 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:02.846469    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw" event={"ID":"24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc","Type":"ContainerDied","Data":"6783f7cdcbe128186eda6e074b4612c39716d915a784f39c992eb48b2e66d093"}
Oct 29 22:30:02.850338 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:02.846699    4559 scope.go:122] "RemoveContainer" containerID="6783f7cdcbe128186eda6e074b4612c39716d915a784f39c992eb48b2e66d093"
Oct 29 22:30:03.855993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:03.855954    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b" event={"ID":"4edf698a-ccad-4baf-a7dd-0e91e2f2409a","Type":"ContainerStarted","Data":"d4de9814edb978e3b00033343d46343b74739c03a10b1a54b65451b4ed9daf07"}
Oct 29 22:30:03.858426 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:03.858393    4559 generic.go:356] "Generic (PLEG): container finished" podID="24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc" containerID="d5b2036fce1c3b00bcddcca744de19592b9153808e9af821cf99466162cf86bc" exitCode=60
Oct 29 22:30:03.858561 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:03.858465    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw" event={"ID":"24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc","Type":"ContainerDied","Data":"d5b2036fce1c3b00bcddcca744de19592b9153808e9af821cf99466162cf86bc"}
Oct 29 22:30:03.858561 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:03.858516    4559 scope.go:122] "RemoveContainer" containerID="6783f7cdcbe128186eda6e074b4612c39716d915a784f39c992eb48b2e66d093"
Oct 29 22:30:03.858751 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:03.858729    4559 scope.go:122] "RemoveContainer" containerID="d5b2036fce1c3b00bcddcca744de19592b9153808e9af821cf99466162cf86bc"
Oct 29 22:30:03.859117 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:30:03.859095    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888550-s2xhw_odh-ai-gateway-infra(24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw" podUID="24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc"
Oct 29 22:30:03.860995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:03.860973    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687" event={"ID":"a29150a5-5b05-493e-9968-cf13fd43b2d6","Type":"ContainerStarted","Data":"449ad751a80220f0419e88f11c0850c741d913a5641e9c609bf5f85fe5abe3be"}
Oct 29 22:30:04.905353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:04.905313    4559 generic.go:356] "Generic (PLEG): container finished" podID="4edf698a-ccad-4baf-a7dd-0e91e2f2409a" containerID="d4de9814edb978e3b00033343d46343b74739c03a10b1a54b65451b4ed9daf07" exitCode=60
Oct 29 22:30:04.906079 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:04.906048    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b" event={"ID":"4edf698a-ccad-4baf-a7dd-0e91e2f2409a","Type":"ContainerDied","Data":"d4de9814edb978e3b00033343d46343b74739c03a10b1a54b65451b4ed9daf07"}
Oct 29 22:30:04.923834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:04.915499    4559 scope.go:122] "RemoveContainer" containerID="d4de9814edb978e3b00033343d46343b74739c03a10b1a54b65451b4ed9daf07"
Oct 29 22:30:04.923834 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:30:04.920750    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b_odh-ai-gateway-infra(4edf698a-ccad-4baf-a7dd-0e91e2f2409a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b" podUID="4edf698a-ccad-4baf-a7dd-0e91e2f2409a"
Oct 29 22:30:04.923834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:04.921020    4559 scope.go:122] "RemoveContainer" containerID="a915a55281823b5d1c573f431e98b02079eb16bb4c73fd38a7cf0658701e9097"
Oct 29 22:30:04.933418 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:04.933392    4559 scope.go:122] "RemoveContainer" containerID="d5b2036fce1c3b00bcddcca744de19592b9153808e9af821cf99466162cf86bc"
Oct 29 22:30:04.933757 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:30:04.933732    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888550-s2xhw_odh-ai-gateway-infra(24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw" podUID="24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc"
Oct 29 22:30:04.936174 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:04.936143    4559 generic.go:356] "Generic (PLEG): container finished" podID="a29150a5-5b05-493e-9968-cf13fd43b2d6" containerID="449ad751a80220f0419e88f11c0850c741d913a5641e9c609bf5f85fe5abe3be" exitCode=60
Oct 29 22:30:04.936279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:04.936211    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687" event={"ID":"a29150a5-5b05-493e-9968-cf13fd43b2d6","Type":"ContainerDied","Data":"449ad751a80220f0419e88f11c0850c741d913a5641e9c609bf5f85fe5abe3be"}
Oct 29 22:30:04.936629 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:04.936610    4559 scope.go:122] "RemoveContainer" containerID="449ad751a80220f0419e88f11c0850c741d913a5641e9c609bf5f85fe5abe3be"
Oct 29 22:30:04.936987 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:30:04.936961    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687_odh-ai-gateway-infra(a29150a5-5b05-493e-9968-cf13fd43b2d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687" podUID="a29150a5-5b05-493e-9968-cf13fd43b2d6"
Oct 29 22:30:04.944385 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:04.944363    4559 scope.go:122] "RemoveContainer" containerID="0b14344309e0d37e341c0c70f0cd851396b1227d60be1952cb510d19d0a08beb"
Oct 29 22:30:05.947112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:05.947077    4559 scope.go:122] "RemoveContainer" containerID="d4de9814edb978e3b00033343d46343b74739c03a10b1a54b65451b4ed9daf07"
Oct 29 22:30:05.947599 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:30:05.947421    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b_odh-ai-gateway-infra(4edf698a-ccad-4baf-a7dd-0e91e2f2409a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b" podUID="4edf698a-ccad-4baf-a7dd-0e91e2f2409a"
Oct 29 22:30:05.949067 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:05.949047    4559 scope.go:122] "RemoveContainer" containerID="449ad751a80220f0419e88f11c0850c741d913a5641e9c609bf5f85fe5abe3be"
Oct 29 22:30:05.949332 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:30:05.949307    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687_odh-ai-gateway-infra(a29150a5-5b05-493e-9968-cf13fd43b2d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687" podUID="a29150a5-5b05-493e-9968-cf13fd43b2d6"
Oct 29 22:30:13.695468 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:30:13.695426    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:30:15.220471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:15.220439    4559 scope.go:122] "RemoveContainer" containerID="d4de9814edb978e3b00033343d46343b74739c03a10b1a54b65451b4ed9daf07"
Oct 29 22:30:16.038775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:16.038731    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b" event={"ID":"4edf698a-ccad-4baf-a7dd-0e91e2f2409a","Type":"ContainerStarted","Data":"ceae76ea4a09cf307bc64c22416c07ff7227520a8e48f2908e019ddcb98994d9"}
Oct 29 22:30:16.220781 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:16.220746    4559 scope.go:122] "RemoveContainer" containerID="449ad751a80220f0419e88f11c0850c741d913a5641e9c609bf5f85fe5abe3be"
Oct 29 22:30:17.050760 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:17.050719    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687" event={"ID":"a29150a5-5b05-493e-9968-cf13fd43b2d6","Type":"ContainerStarted","Data":"4d7ac06d55c9f0446c43e02d08d589ed3a8414b7dfa5c81a13390bf40aa4850e"}
Oct 29 22:30:17.052909 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:17.052881    4559 generic.go:356] "Generic (PLEG): container finished" podID="4edf698a-ccad-4baf-a7dd-0e91e2f2409a" containerID="ceae76ea4a09cf307bc64c22416c07ff7227520a8e48f2908e019ddcb98994d9" exitCode=60
Oct 29 22:30:17.053046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:17.052960    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b" event={"ID":"4edf698a-ccad-4baf-a7dd-0e91e2f2409a","Type":"ContainerDied","Data":"ceae76ea4a09cf307bc64c22416c07ff7227520a8e48f2908e019ddcb98994d9"}
Oct 29 22:30:17.053046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:17.053031    4559 scope.go:122] "RemoveContainer" containerID="d4de9814edb978e3b00033343d46343b74739c03a10b1a54b65451b4ed9daf07"
Oct 29 22:30:17.053407 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:17.053389    4559 scope.go:122] "RemoveContainer" containerID="ceae76ea4a09cf307bc64c22416c07ff7227520a8e48f2908e019ddcb98994d9"
Oct 29 22:30:17.053743 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:30:17.053717    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b_odh-ai-gateway-infra(4edf698a-ccad-4baf-a7dd-0e91e2f2409a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b" podUID="4edf698a-ccad-4baf-a7dd-0e91e2f2409a"
Oct 29 22:30:17.087933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:17.087897    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b"]
Oct 29 22:30:17.220369 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:17.220340    4559 scope.go:122] "RemoveContainer" containerID="d5b2036fce1c3b00bcddcca744de19592b9153808e9af821cf99466162cf86bc"
Oct 29 22:30:18.083313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:18.082182    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw" event={"ID":"24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc","Type":"ContainerStarted","Data":"f00b79ce6e2b6b8ae0165b57aad7c150fb221be612b31cfef3fb99ceda24d698"}
Oct 29 22:30:18.087418 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:18.087383    4559 generic.go:356] "Generic (PLEG): container finished" podID="a29150a5-5b05-493e-9968-cf13fd43b2d6" containerID="4d7ac06d55c9f0446c43e02d08d589ed3a8414b7dfa5c81a13390bf40aa4850e" exitCode=60
Oct 29 22:30:18.088073 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:18.088031    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687" event={"ID":"a29150a5-5b05-493e-9968-cf13fd43b2d6","Type":"ContainerDied","Data":"4d7ac06d55c9f0446c43e02d08d589ed3a8414b7dfa5c81a13390bf40aa4850e"}
Oct 29 22:30:18.088317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:18.088285    4559 scope.go:122] "RemoveContainer" containerID="449ad751a80220f0419e88f11c0850c741d913a5641e9c609bf5f85fe5abe3be"
Oct 29 22:30:18.089057 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:18.089033    4559 scope.go:122] "RemoveContainer" containerID="4d7ac06d55c9f0446c43e02d08d589ed3a8414b7dfa5c81a13390bf40aa4850e"
Oct 29 22:30:18.089732 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:30:18.089693    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687_odh-ai-gateway-infra(a29150a5-5b05-493e-9968-cf13fd43b2d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687" podUID="a29150a5-5b05-493e-9968-cf13fd43b2d6"
Oct 29 22:30:18.135856 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:18.135823    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687"]
Oct 29 22:30:18.440628 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:18.440335    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b"
Oct 29 22:30:18.560571 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:18.558531    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4edf698a-ccad-4baf-a7dd-0e91e2f2409a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4edf698a-ccad-4baf-a7dd-0e91e2f2409a-maas-api-service-ca\") pod \"4edf698a-ccad-4baf-a7dd-0e91e2f2409a\" (UID: \"4edf698a-ccad-4baf-a7dd-0e91e2f2409a\") "
Oct 29 22:30:18.560571 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:18.558692    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4edf698a-ccad-4baf-a7dd-0e91e2f2409a-kube-api-access-xvnb4\" (UniqueName: \"kubernetes.io/projected/4edf698a-ccad-4baf-a7dd-0e91e2f2409a-kube-api-access-xvnb4\") pod \"4edf698a-ccad-4baf-a7dd-0e91e2f2409a\" (UID: \"4edf698a-ccad-4baf-a7dd-0e91e2f2409a\") "
Oct 29 22:30:18.561081 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:18.561028    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4edf698a-ccad-4baf-a7dd-0e91e2f2409a-maas-api-service-ca" pod "4edf698a-ccad-4baf-a7dd-0e91e2f2409a" (UID: "4edf698a-ccad-4baf-a7dd-0e91e2f2409a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:30:18.607551 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:18.607435    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4edf698a-ccad-4baf-a7dd-0e91e2f2409a-kube-api-access-xvnb4" pod "4edf698a-ccad-4baf-a7dd-0e91e2f2409a" (UID: "4edf698a-ccad-4baf-a7dd-0e91e2f2409a"). InnerVolumeSpecName "kube-api-access-xvnb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:30:18.660752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:18.660711    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4edf698a-ccad-4baf-a7dd-0e91e2f2409a-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:30:18.660752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:18.660753    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvnb4\" (UniqueName: \"kubernetes.io/projected/4edf698a-ccad-4baf-a7dd-0e91e2f2409a-kube-api-access-xvnb4\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:30:19.159287 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:19.159242    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw"]
Oct 29 22:30:19.244894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:19.229284    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b" event={"ID":"4edf698a-ccad-4baf-a7dd-0e91e2f2409a","Type":"ContainerDied","Data":"8a27f549707d0fa60eeb35c4fc932d9e7c83880e8510def647dd167742ad65ad"}
Oct 29 22:30:19.244894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:19.229764    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b"
Oct 29 22:30:19.246198 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:19.246172    4559 scope.go:122] "RemoveContainer" containerID="ceae76ea4a09cf307bc64c22416c07ff7227520a8e48f2908e019ddcb98994d9"
Oct 29 22:30:19.368200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:19.367661    4559 generic.go:356] "Generic (PLEG): container finished" podID="24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc" containerID="f00b79ce6e2b6b8ae0165b57aad7c150fb221be612b31cfef3fb99ceda24d698" exitCode=60
Oct 29 22:30:19.368200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:19.367720    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw" event={"ID":"24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc","Type":"ContainerDied","Data":"f00b79ce6e2b6b8ae0165b57aad7c150fb221be612b31cfef3fb99ceda24d698"}
Oct 29 22:30:19.538337 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:19.538259    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b"]
Oct 29 22:30:19.550087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:19.539963    4559 scope.go:122] "RemoveContainer" containerID="d5b2036fce1c3b00bcddcca744de19592b9153808e9af821cf99466162cf86bc"
Oct 29 22:30:19.564133 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:19.564092    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888550-x7d9b"]
Oct 29 22:30:20.224514 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.224488    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687"
Oct 29 22:30:20.274503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.274469    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw"
Oct 29 22:30:20.318352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.318313    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29150a5-5b05-493e-9968-cf13fd43b2d6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29150a5-5b05-493e-9968-cf13fd43b2d6-maas-api-service-ca\") pod \"a29150a5-5b05-493e-9968-cf13fd43b2d6\" (UID: \"a29150a5-5b05-493e-9968-cf13fd43b2d6\") "
Oct 29 22:30:20.318553 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.318381    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc-kube-api-access-x9vhq\" (UniqueName: \"kubernetes.io/projected/24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc-kube-api-access-x9vhq\") pod \"24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc\" (UID: \"24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc\") "
Oct 29 22:30:20.318981 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.318952    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29150a5-5b05-493e-9968-cf13fd43b2d6-maas-api-service-ca" pod "a29150a5-5b05-493e-9968-cf13fd43b2d6" (UID: "a29150a5-5b05-493e-9968-cf13fd43b2d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:30:20.319102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.319074    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc-maas-api-service-ca\") pod \"24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc\" (UID: \"24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc\") "
Oct 29 22:30:20.319245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.319226    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29150a5-5b05-493e-9968-cf13fd43b2d6-kube-api-access-7rxc6\" (UniqueName: \"kubernetes.io/projected/a29150a5-5b05-493e-9968-cf13fd43b2d6-kube-api-access-7rxc6\") pod \"a29150a5-5b05-493e-9968-cf13fd43b2d6\" (UID: \"a29150a5-5b05-493e-9968-cf13fd43b2d6\") "
Oct 29 22:30:20.320022 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.319999    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29150a5-5b05-493e-9968-cf13fd43b2d6-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:30:20.320457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.320434    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc-maas-api-service-ca" pod "24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc" (UID: "24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:30:20.323581 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.323543    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc-kube-api-access-x9vhq" pod "24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc" (UID: "24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc"). InnerVolumeSpecName "kube-api-access-x9vhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:30:20.324738 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.324712    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29150a5-5b05-493e-9968-cf13fd43b2d6-kube-api-access-7rxc6" pod "a29150a5-5b05-493e-9968-cf13fd43b2d6" (UID: "a29150a5-5b05-493e-9968-cf13fd43b2d6"). InnerVolumeSpecName "kube-api-access-7rxc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:30:20.379049 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.379018    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687"
Oct 29 22:30:20.379241 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.379046    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687" event={"ID":"a29150a5-5b05-493e-9968-cf13fd43b2d6","Type":"ContainerDied","Data":"12393468b4a531bcbd1bd2307a1c8776a5a99b5807eaa5f6287e3156a8d0fde7"}
Oct 29 22:30:20.379241 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.379104    4559 scope.go:122] "RemoveContainer" containerID="4d7ac06d55c9f0446c43e02d08d589ed3a8414b7dfa5c81a13390bf40aa4850e"
Oct 29 22:30:20.382390 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.382365    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw"
Oct 29 22:30:20.382534 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.382364    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw" event={"ID":"24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc","Type":"ContainerDied","Data":"aa7c9a940f4fa5259968184e975db06b5c44d5279faf7d338f2f9209bd5ec04f"}
Oct 29 22:30:20.390022 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.390002    4559 scope.go:122] "RemoveContainer" containerID="f00b79ce6e2b6b8ae0165b57aad7c150fb221be612b31cfef3fb99ceda24d698"
Oct 29 22:30:20.408280 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.408250    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"]
Oct 29 22:30:20.408787 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.408774    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4edf698a-ccad-4baf-a7dd-0e91e2f2409a" containerName="cleanup"
Oct 29 22:30:20.408865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.408788    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4edf698a-ccad-4baf-a7dd-0e91e2f2409a" containerName="cleanup"
Oct 29 22:30:20.408865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.408798    4559 cpu_manager.go:414] "containerMap: removing container" podUID="24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc" containerName="cleanup"
Oct 29 22:30:20.408865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.408804    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc" containerName="cleanup"
Oct 29 22:30:20.408865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.408841    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a29150a5-5b05-493e-9968-cf13fd43b2d6" containerName="cleanup"
Oct 29 22:30:20.408865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.408845    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29150a5-5b05-493e-9968-cf13fd43b2d6" containerName="cleanup"
Oct 29 22:30:20.408865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.408863    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a29150a5-5b05-493e-9968-cf13fd43b2d6" containerName="cleanup"
Oct 29 22:30:20.408865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.408868    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29150a5-5b05-493e-9968-cf13fd43b2d6" containerName="cleanup"
Oct 29 22:30:20.409201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.408876    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4edf698a-ccad-4baf-a7dd-0e91e2f2409a" containerName="cleanup"
Oct 29 22:30:20.409201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.408881    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4edf698a-ccad-4baf-a7dd-0e91e2f2409a" containerName="cleanup"
Oct 29 22:30:20.409201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.408897    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a29150a5-5b05-493e-9968-cf13fd43b2d6" containerName="cleanup"
Oct 29 22:30:20.409201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.408901    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29150a5-5b05-493e-9968-cf13fd43b2d6" containerName="cleanup"
Oct 29 22:30:20.409201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.408911    4559 cpu_manager.go:414] "containerMap: removing container" podUID="24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc" containerName="cleanup"
Oct 29 22:30:20.409201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.408915    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc" containerName="cleanup"
Oct 29 22:30:20.409201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.409011    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4edf698a-ccad-4baf-a7dd-0e91e2f2409a" containerName="cleanup"
Oct 29 22:30:20.409201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.409019    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc" containerName="cleanup"
Oct 29 22:30:20.409201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.409025    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29150a5-5b05-493e-9968-cf13fd43b2d6" containerName="cleanup"
Oct 29 22:30:20.409201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.409031    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29150a5-5b05-493e-9968-cf13fd43b2d6" containerName="cleanup"
Oct 29 22:30:20.409201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.409036    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4edf698a-ccad-4baf-a7dd-0e91e2f2409a" containerName="cleanup"
Oct 29 22:30:20.409201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.409042    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4edf698a-ccad-4baf-a7dd-0e91e2f2409a" containerName="cleanup"
Oct 29 22:30:20.409201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.409049    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc" containerName="cleanup"
Oct 29 22:30:20.409201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.409054    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc" containerName="cleanup"
Oct 29 22:30:20.414756 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.414732    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"
Oct 29 22:30:20.421090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.421065    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:30:20.421090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.421089    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rxc6\" (UniqueName: \"kubernetes.io/projected/a29150a5-5b05-493e-9968-cf13fd43b2d6-kube-api-access-7rxc6\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:30:20.421303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.421100    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9vhq\" (UniqueName: \"kubernetes.io/projected/24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc-kube-api-access-x9vhq\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:30:20.431088 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.431063    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:30:20.431264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.431136    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:30:20.433141 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.432892    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"]
Oct 29 22:30:20.440757 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.440439    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw"]
Oct 29 22:30:20.447302 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.447259    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888550-s2xhw"]
Oct 29 22:30:20.486724 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.486681    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687"]
Oct 29 22:30:20.490183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.490153    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888550-8f687"]
Oct 29 22:30:20.521901 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.521861    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmb9r\" (UniqueName: \"kubernetes.io/projected/1462bbfd-3301-4194-87ce-2961af36892a-kube-api-access-mmb9r\") pod \"maas-api-revoke-sub-dea14161-shtcb\" (UID: \"1462bbfd-3301-4194-87ce-2961af36892a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"
Oct 29 22:30:20.522098 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.521977    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1462bbfd-3301-4194-87ce-2961af36892a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dea14161-shtcb\" (UID: \"1462bbfd-3301-4194-87ce-2961af36892a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"
Oct 29 22:30:20.522098 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.522032    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1462bbfd-3301-4194-87ce-2961af36892a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dea14161-shtcb\" (UID: \"1462bbfd-3301-4194-87ce-2961af36892a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"
Oct 29 22:30:20.622866 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.622828    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1462bbfd-3301-4194-87ce-2961af36892a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dea14161-shtcb\" (UID: \"1462bbfd-3301-4194-87ce-2961af36892a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"
Oct 29 22:30:20.623038 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.622887    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1462bbfd-3301-4194-87ce-2961af36892a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dea14161-shtcb\" (UID: \"1462bbfd-3301-4194-87ce-2961af36892a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"
Oct 29 22:30:20.623078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.623034    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmb9r\" (UniqueName: \"kubernetes.io/projected/1462bbfd-3301-4194-87ce-2961af36892a-kube-api-access-mmb9r\") pod \"maas-api-revoke-sub-dea14161-shtcb\" (UID: \"1462bbfd-3301-4194-87ce-2961af36892a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"
Oct 29 22:30:20.623722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.623702    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1462bbfd-3301-4194-87ce-2961af36892a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dea14161-shtcb\" (UID: \"1462bbfd-3301-4194-87ce-2961af36892a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"
Oct 29 22:30:20.625546 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.625521    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1462bbfd-3301-4194-87ce-2961af36892a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dea14161-shtcb\" (UID: \"1462bbfd-3301-4194-87ce-2961af36892a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"
Oct 29 22:30:20.645307 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.645271    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmb9r\" (UniqueName: \"kubernetes.io/projected/1462bbfd-3301-4194-87ce-2961af36892a-kube-api-access-mmb9r\") pod \"maas-api-revoke-sub-dea14161-shtcb\" (UID: \"1462bbfd-3301-4194-87ce-2961af36892a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"
Oct 29 22:30:20.728930 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:20.728894    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"
Oct 29 22:30:21.036410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:21.036380    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"]
Oct 29 22:30:21.036931 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:30:21.036903    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1462bbfd_3301_4194_87ce_2961af36892a.slice/crio-d84eedebbfcbc2894373c4d3ab50d1d5706f9f66b6bf2043d0b179c17de42eca WatchSource:0}: Error finding container d84eedebbfcbc2894373c4d3ab50d1d5706f9f66b6bf2043d0b179c17de42eca: Status 404 returned error can't find the container with id d84eedebbfcbc2894373c4d3ab50d1d5706f9f66b6bf2043d0b179c17de42eca
Oct 29 22:30:21.228076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:21.228039    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc" path="/var/lib/kubelet/pods/24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc/volumes"
Oct 29 22:30:21.228490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:21.228476    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4edf698a-ccad-4baf-a7dd-0e91e2f2409a" path="/var/lib/kubelet/pods/4edf698a-ccad-4baf-a7dd-0e91e2f2409a/volumes"
Oct 29 22:30:21.228863 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:21.228848    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29150a5-5b05-493e-9968-cf13fd43b2d6" path="/var/lib/kubelet/pods/a29150a5-5b05-493e-9968-cf13fd43b2d6/volumes"
Oct 29 22:30:21.390062 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:21.390027    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb" event={"ID":"1462bbfd-3301-4194-87ce-2961af36892a","Type":"ContainerStarted","Data":"f4bdcb67fe7e600088f0d4a3431669694fd029436f4f7ce9c7e5320ed385b749"}
Oct 29 22:30:21.390227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:21.390090    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb" event={"ID":"1462bbfd-3301-4194-87ce-2961af36892a","Type":"ContainerStarted","Data":"d84eedebbfcbc2894373c4d3ab50d1d5706f9f66b6bf2043d0b179c17de42eca"}
Oct 29 22:30:21.433625 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:21.433547    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb" podStartSLOduration=1.433528305 podStartE2EDuration="1.433528305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:21.430957478 +0000 UTC m=+960.342401031" watchObservedRunningTime="2026-10-29 22:30:21.433528305 +0000 UTC m=+960.344971862"
Oct 29 22:30:22.398743 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:22.398708    4559 generic.go:356] "Generic (PLEG): container finished" podID="1462bbfd-3301-4194-87ce-2961af36892a" containerID="f4bdcb67fe7e600088f0d4a3431669694fd029436f4f7ce9c7e5320ed385b749" exitCode=0
Oct 29 22:30:22.399133 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:22.398793    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb" event={"ID":"1462bbfd-3301-4194-87ce-2961af36892a","Type":"ContainerDied","Data":"f4bdcb67fe7e600088f0d4a3431669694fd029436f4f7ce9c7e5320ed385b749"}
Oct 29 22:30:23.572918 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:23.572881    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"
Oct 29 22:30:23.665920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:23.665804    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1462bbfd-3301-4194-87ce-2961af36892a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1462bbfd-3301-4194-87ce-2961af36892a-maas-api-xks-ca\") pod \"1462bbfd-3301-4194-87ce-2961af36892a\" (UID: \"1462bbfd-3301-4194-87ce-2961af36892a\") "
Oct 29 22:30:23.666086 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:23.666014    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1462bbfd-3301-4194-87ce-2961af36892a-kube-api-access-mmb9r\" (UniqueName: \"kubernetes.io/projected/1462bbfd-3301-4194-87ce-2961af36892a-kube-api-access-mmb9r\") pod \"1462bbfd-3301-4194-87ce-2961af36892a\" (UID: \"1462bbfd-3301-4194-87ce-2961af36892a\") "
Oct 29 22:30:23.666086 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:23.666042    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1462bbfd-3301-4194-87ce-2961af36892a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1462bbfd-3301-4194-87ce-2961af36892a-maas-api-openshift-ca\") pod \"1462bbfd-3301-4194-87ce-2961af36892a\" (UID: \"1462bbfd-3301-4194-87ce-2961af36892a\") "
Oct 29 22:30:23.666547 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:23.666518    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1462bbfd-3301-4194-87ce-2961af36892a-maas-api-openshift-ca" pod "1462bbfd-3301-4194-87ce-2961af36892a" (UID: "1462bbfd-3301-4194-87ce-2961af36892a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:30:23.668208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:23.668179    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1462bbfd-3301-4194-87ce-2961af36892a-maas-api-xks-ca" pod "1462bbfd-3301-4194-87ce-2961af36892a" (UID: "1462bbfd-3301-4194-87ce-2961af36892a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:30:23.668347 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:23.668239    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1462bbfd-3301-4194-87ce-2961af36892a-kube-api-access-mmb9r" pod "1462bbfd-3301-4194-87ce-2961af36892a" (UID: "1462bbfd-3301-4194-87ce-2961af36892a"). InnerVolumeSpecName "kube-api-access-mmb9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:30:23.767737 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:23.767693    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1462bbfd-3301-4194-87ce-2961af36892a-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:30:23.767737 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:23.767724    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmb9r\" (UniqueName: \"kubernetes.io/projected/1462bbfd-3301-4194-87ce-2961af36892a-kube-api-access-mmb9r\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:30:23.767737 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:23.767734    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1462bbfd-3301-4194-87ce-2961af36892a-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:30:24.412448 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:24.412415    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"
Oct 29 22:30:24.412617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:24.412491    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb" event={"ID":"1462bbfd-3301-4194-87ce-2961af36892a","Type":"ContainerDied","Data":"d84eedebbfcbc2894373c4d3ab50d1d5706f9f66b6bf2043d0b179c17de42eca"}
Oct 29 22:30:24.412617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:24.412537    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d84eedebbfcbc2894373c4d3ab50d1d5706f9f66b6bf2043d0b179c17de42eca"
Oct 29 22:30:33.012447 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:33.012410    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:30:35.926240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.926205    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"]
Oct 29 22:30:35.926920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.926900    4559 cpu_manager.go:414] "containerMap: removing container" podUID="1462bbfd-3301-4194-87ce-2961af36892a" containerName="revoke-keys"
Oct 29 22:30:35.926920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.926920    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1462bbfd-3301-4194-87ce-2961af36892a" containerName="revoke-keys"
Oct 29 22:30:35.927087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.926938    4559 cpu_manager.go:414] "containerMap: removing container" podUID="24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc" containerName="cleanup"
Oct 29 22:30:35.927087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.926946    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f4d3b5-a216-4d2a-b8eb-f3d09326b6dc" containerName="cleanup"
Oct 29 22:30:35.927087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.926972    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4edf698a-ccad-4baf-a7dd-0e91e2f2409a" containerName="cleanup"
Oct 29 22:30:35.927087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.926979    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4edf698a-ccad-4baf-a7dd-0e91e2f2409a" containerName="cleanup"
Oct 29 22:30:35.927241 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.927119    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="1462bbfd-3301-4194-87ce-2961af36892a" containerName="revoke-keys"
Oct 29 22:30:35.927241 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.927136    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29150a5-5b05-493e-9968-cf13fd43b2d6" containerName="cleanup"
Oct 29 22:30:35.931788 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.931757    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:35.954519 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.954487    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"]
Oct 29 22:30:35.964920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.964870    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-08ebee\"/\"distinct-2-w4-08ebee-kserve-self-signed-certs\""
Oct 29 22:30:35.985123 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.985078    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xx2n\" (UniqueName: \"kubernetes.io/projected/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-kube-api-access-7xx2n\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:35.985123 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.985119    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-tmp-dir\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:35.985422 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.985310    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-dshm\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:35.985422 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.985385    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-model-cache\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:35.985564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.985462    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-tls-certs\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:35.985564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:35.985546    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-home\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:36.035925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.035888    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"]
Oct 29 22:30:36.040656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.040630    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.072711 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.072653    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"]
Oct 29 22:30:36.086548 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.086516    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.086749 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.086565    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.086749 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.086626    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xx2n\" (UniqueName: \"kubernetes.io/projected/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-kube-api-access-7xx2n\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:36.086749 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.086648    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.086749 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.086671    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-tmp-dir\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:36.086997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.086755    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ebc209-264e-49db-87fd-ca9c789386b4-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.086997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.086833    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.087097 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.087072    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-tmp-dir\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:36.087299 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.087280    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-dshm\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:36.087619 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.087600    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ebc209-264e-49db-87fd-ca9c789386b4-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.087702 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.087664    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-model-cache\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:36.088149 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.088125    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-tls-certs\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:36.089306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.089248    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-model-cache\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:36.089839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.089791    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"]
Oct 29 22:30:36.090087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.089925    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.090154 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.090110    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5xc6\" (UniqueName: \"kubernetes.io/projected/62ebc209-264e-49db-87fd-ca9c789386b4-kube-api-access-b5xc6\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.090154 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.090143    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ebc209-264e-49db-87fd-ca9c789386b4-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.090251 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.090178    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-home\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:36.090901 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.090876    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-home\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:36.096694 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.096672    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.105680 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.105622    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-dshm\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:36.114801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.109618    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xx2n\" (UniqueName: \"kubernetes.io/projected/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-kube-api-access-7xx2n\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:36.143793 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.119667    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"]
Oct 29 22:30:36.155837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.148713    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-tls-certs\") pod \"distinct-2-w4-08ebee-kserve-657686d8c-82jmg\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:36.226852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.193610    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.248029 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.212286    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.252856 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.252787    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:36.257054 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.257018    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5xc6\" (UniqueName: \"kubernetes.io/projected/62ebc209-264e-49db-87fd-ca9c789386b4-kube-api-access-b5xc6\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.257271 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.257253    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ebc209-264e-49db-87fd-ca9c789386b4-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.257496 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.257481    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.257639 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.257620    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.263290 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.257791    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.263661 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.263574    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ebc209-264e-49db-87fd-ca9c789386b4-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.263904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.263886    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.264138 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.264116    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ebc209-264e-49db-87fd-ca9c789386b4-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.276654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.276548    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ebc209-264e-49db-87fd-ca9c789386b4-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.281369 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.280066    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.283666 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.283538    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.283803 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.283728    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.284800 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.284774    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ebc209-264e-49db-87fd-ca9c789386b4-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.286387 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.286364    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.289141 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.289112    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5xc6\" (UniqueName: \"kubernetes.io/projected/62ebc209-264e-49db-87fd-ca9c789386b4-kube-api-access-b5xc6\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.290712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.290681    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ebc209-264e-49db-87fd-ca9c789386b4-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.358350 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.358313    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:36.366485 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.365350    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.366485 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.365399    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.366485 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.365430    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.366485 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.365453    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.366485 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.365488    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.366485 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.365575    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.366485 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.365731    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4xxr\" (UniqueName: \"kubernetes.io/projected/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-kube-api-access-j4xxr\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.366485 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.365831    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.366485 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.365864    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.468905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.468642    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4xxr\" (UniqueName: \"kubernetes.io/projected/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-kube-api-access-j4xxr\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.468905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.468734    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.468905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.468765    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.468905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.468856    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.468905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.468884    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.469284 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.468916    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.469284 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.468940    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.469284 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.468974    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.469284 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.469055    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.471080 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.469965    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.471080 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.470746    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.471312 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.471123    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.476671 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.476571    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.476671 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.476644    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.480865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.477551    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.482623 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.481627    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.493038 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.493010    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.493658 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.493630    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4xxr\" (UniqueName: \"kubernetes.io/projected/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-kube-api-access-j4xxr\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.512698 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.512243    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:36.595989 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.595943    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"]
Oct 29 22:30:36.750750 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.750723    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"]
Oct 29 22:30:36.752130 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:30:36.752084    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ebc209_264e_49db_87fd_ca9c789386b4.slice/crio-8eedd185a5e7a0ecfd2730cb0fb55be2896e837ba37b19eba7ecdcbbdb7aae61 WatchSource:0}: Error finding container 8eedd185a5e7a0ecfd2730cb0fb55be2896e837ba37b19eba7ecdcbbdb7aae61: Status 404 returned error can't find the container with id 8eedd185a5e7a0ecfd2730cb0fb55be2896e837ba37b19eba7ecdcbbdb7aae61
Oct 29 22:30:36.755144 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.755113    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:30:36.755231 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.755181    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:30:36.755231 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.755221    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:30:36.832472 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.832404    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"]
Oct 29 22:30:36.834937 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:30:36.834901    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aec0aeb_95b6_4ea8_9e61_c0a97e6b1518.slice/crio-0da01749f46e18ccfc275b9cc77a4484c3166d4325ea47b0581d1584552b3f90 WatchSource:0}: Error finding container 0da01749f46e18ccfc275b9cc77a4484c3166d4325ea47b0581d1584552b3f90: Status 404 returned error can't find the container with id 0da01749f46e18ccfc275b9cc77a4484c3166d4325ea47b0581d1584552b3f90
Oct 29 22:30:36.837853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.837780    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:30:36.837940 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.837912    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:30:36.837988 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:36.837956    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:30:37.551977 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:37.551928    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv" event={"ID":"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518","Type":"ContainerStarted","Data":"b908886771c48e6baeeb8200680778da8bec333b878c5ffd52ffac00bfb9532a"}
Oct 29 22:30:37.552844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:37.552646    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv" event={"ID":"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518","Type":"ContainerStarted","Data":"0da01749f46e18ccfc275b9cc77a4484c3166d4325ea47b0581d1584552b3f90"}
Oct 29 22:30:37.566838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:37.563802    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t" event={"ID":"62ebc209-264e-49db-87fd-ca9c789386b4","Type":"ContainerStarted","Data":"c4fe2ada8cd77fbef664dac65577b1068589f03db9641ae83e3ab5adddfde0c1"}
Oct 29 22:30:37.566838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:37.563897    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t" event={"ID":"62ebc209-264e-49db-87fd-ca9c789386b4","Type":"ContainerStarted","Data":"8eedd185a5e7a0ecfd2730cb0fb55be2896e837ba37b19eba7ecdcbbdb7aae61"}
Oct 29 22:30:37.590240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:37.590204    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" event={"ID":"daabb28c-a3d4-43f5-85d7-b09141c8a4fd","Type":"ContainerStarted","Data":"91b6aac3f7c570c7478709b1d31627e07a983b81a8d2f500ab49ba84da2c7a1b"}
Oct 29 22:30:37.590240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:37.590244    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" event={"ID":"daabb28c-a3d4-43f5-85d7-b09141c8a4fd","Type":"ContainerStarted","Data":"31531d637c4551224ce194f140463dcecec8eb7043a31e3dee43380fde4ab502"}
Oct 29 22:30:37.599475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:37.599390    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv" podStartSLOduration=1.59937385 podStartE2EDuration="1.59937385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:37.596227126 +0000 UTC m=+976.507670691" watchObservedRunningTime="2026-10-29 22:30:37.59937385 +0000 UTC m=+976.510817404"
Oct 29 22:30:37.689550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:37.689473    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podStartSLOduration=2.6894576900000002 podStartE2EDuration="2.68945769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:37.683051099 +0000 UTC m=+976.594494678" watchObservedRunningTime="2026-10-29 22:30:37.68945769 +0000 UTC m=+976.600901654"
Oct 29 22:30:37.728117 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:37.728058    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t" podStartSLOduration=1.728039931 podStartE2EDuration="1.728039931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:37.724468307 +0000 UTC m=+976.635911872" watchObservedRunningTime="2026-10-29 22:30:37.728039931 +0000 UTC m=+976.639483485"
Oct 29 22:30:38.257945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:38.257904    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:38.359906 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:38.359871    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:38.361184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:38.361156    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:30:38.361411 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:38.361383    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t" podUID="62ebc209-264e-49db-87fd-ca9c789386b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:30:38.515628 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:38.515368    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:38.516035 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:38.516014    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:30:38.516144 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:38.516059    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv" podUID="4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:30:39.359836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:39.359239    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:30:39.359836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:39.359301    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t" podUID="62ebc209-264e-49db-87fd-ca9c789386b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:30:39.514694 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:39.514656    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:30:39.514861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:39.514712    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv" podUID="4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:30:40.367403 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:40.367364    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:30:40.368052 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:40.368015    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t" podUID="62ebc209-264e-49db-87fd-ca9c789386b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:30:40.514932 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:40.514895    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:30:40.515136 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:40.514960    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv" podUID="4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:30:41.363296 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:41.363253    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:30:41.363670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:41.363635    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t" podUID="62ebc209-264e-49db-87fd-ca9c789386b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:30:41.514632 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:41.514598    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:30:41.515125 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:41.515097    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv" podUID="4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:30:42.364836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:42.364229    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:30:42.364836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:42.364291    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t" podUID="62ebc209-264e-49db-87fd-ca9c789386b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:30:42.514269 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:42.514232    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:30:42.514457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:42.514290    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv" podUID="4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:30:43.359107 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:43.359066    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:30:43.359485 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:43.359136    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t" podUID="62ebc209-264e-49db-87fd-ca9c789386b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:30:43.515170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:43.515129    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:30:43.515361 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:43.515188    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv" podUID="4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:30:43.698187 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:30:43.698149    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:30:44.661832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:44.660823    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:44.661832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:44.661825    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:44.663080 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:44.663028    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:30:44.748138 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:44.748105    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:44.749393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:44.749355    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:44.750757 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:44.750736    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:30:46.257032 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:46.256997    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:46.270454 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:46.270424    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:46.671166 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:46.671132    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:30:50.446836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.446106    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"]
Oct 29 22:30:50.453843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.452199    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.492693 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.492659    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"]
Oct 29 22:30:50.523928 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.523891    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.524192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.524167    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.524411 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.524391    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.524559 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.524541    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220a1952-3291-485e-b6d4-4336b1cd1db3-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.524691 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.524675    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.524907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.524884    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v94tv\" (UniqueName: \"kubernetes.io/projected/220a1952-3291-485e-b6d4-4336b1cd1db3-kube-api-access-v94tv\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.525483 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.525450    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.525677 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.525645    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.525882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.525831    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.631904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.628538    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.631904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.629747    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220a1952-3291-485e-b6d4-4336b1cd1db3-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.631904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.629912    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.631904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.630075    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v94tv\" (UniqueName: \"kubernetes.io/projected/220a1952-3291-485e-b6d4-4336b1cd1db3-kube-api-access-v94tv\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.631904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.630374    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.633144 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.630427    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.633358 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.633332    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.634197 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.634176    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.634465 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.634449    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.650868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.633026    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.650868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.630776    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.650868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.645223    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.650868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.631429    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220a1952-3291-485e-b6d4-4336b1cd1db3-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.650868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.629074    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.673306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.662690    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.692111 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.692056    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v94tv\" (UniqueName: \"kubernetes.io/projected/220a1952-3291-485e-b6d4-4336b1cd1db3-kube-api-access-v94tv\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.733948 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.726430    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.733948 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.726921    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:50.845100 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:50.845061    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:51.721421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:51.721354    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:30:51.751854 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:51.737090    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:30:51.947870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:51.943050    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"]
Oct 29 22:30:51.976922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:51.971088    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:30:52.123134 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:52.118381    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:30:52.123134 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:52.118564    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:30:52.123134 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:52.118663    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:30:52.134834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:52.127680    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:30:52.525865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:52.516956    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:30:52.762843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:52.735351    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:30:52.762843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:52.735710    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:30:52.935934 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:52.935634    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:30:53.196228 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:53.196135    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl" event={"ID":"220a1952-3291-485e-b6d4-4336b1cd1db3","Type":"ContainerStarted","Data":"ce451b6c74fb7c2ccf5cb17f78c6a3a2f5800da4992e7b1ccdd2c0b9c1abe1eb"}
Oct 29 22:30:53.196543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:53.196518    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl" event={"ID":"220a1952-3291-485e-b6d4-4336b1cd1db3","Type":"ContainerStarted","Data":"8a29a65d1a23a3450e66fa59a40ed24f33ff2a3ceb23f5d450a919a4a47d5a4b"}
Oct 29 22:30:53.262385 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:53.262323    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl" podStartSLOduration=3.262303322 podStartE2EDuration="3.262303322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:53.261668827 +0000 UTC m=+992.173112382" watchObservedRunningTime="2026-10-29 22:30:53.262303322 +0000 UTC m=+992.173746876"
Oct 29 22:30:53.846665 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:53.846618    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:30:53.848783 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:53.848749    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:30:53.848936 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:53.848802    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl" podUID="220a1952-3291-485e-b6d4-4336b1cd1db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 22:30:54.846337 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:54.846300    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:30:54.846631 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:54.846598    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl" podUID="220a1952-3291-485e-b6d4-4336b1cd1db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 22:30:55.856315 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:55.856272    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:30:55.863982 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:55.863927    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl" podUID="220a1952-3291-485e-b6d4-4336b1cd1db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 22:30:56.851290 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:56.851253    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:30:56.851597 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:56.851568    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl" podUID="220a1952-3291-485e-b6d4-4336b1cd1db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 22:30:57.850302 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:57.850262    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:30:57.850948 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:57.850914    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl" podUID="220a1952-3291-485e-b6d4-4336b1cd1db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 22:30:58.854769 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:58.854717    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:30:58.855469 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:58.855419    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl" podUID="220a1952-3291-485e-b6d4-4336b1cd1db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 22:30:59.846793 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:59.846752    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:30:59.847020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:30:59.846875    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl" podUID="220a1952-3291-485e-b6d4-4336b1cd1db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 22:31:00.940598 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:00.940561    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:31:00.941768 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:00.941742    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:31:00.944857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:00.944801    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:31:05.512663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.512622    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"]
Oct 29 22:31:05.525575 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.525541    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.550839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.540734    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"]
Oct 29 22:31:05.574832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.563770    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.574832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.563876    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.574832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.564019    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.574832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.564072    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.574832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.564117    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.574832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.564154    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvw8f\" (UniqueName: \"kubernetes.io/projected/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-kube-api-access-jvw8f\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.574832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.564198    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.574832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.564285    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.574832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.564316    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.621640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.621605    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"]
Oct 29 22:31:05.625955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.625929    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.652888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.652850    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"]
Oct 29 22:31:05.665341 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.665302    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.665618 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.665602    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rlfk\" (UniqueName: \"kubernetes.io/projected/b616ef83-09fe-47a2-9d15-4ba5a0126998-kube-api-access-6rlfk\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.665852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.665797    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.666015 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.666001    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.666590 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.666567    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.667175 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.667157    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.667332 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.667317    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvw8f\" (UniqueName: \"kubernetes.io/projected/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-kube-api-access-jvw8f\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.667755 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.667734    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.668219 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.668202    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.668414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.668399    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.668537 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.668524    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.668683 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.668670    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.669062 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.669046    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.669746 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.669727    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.670576 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.670560    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.671510 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.667569    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.671612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.669589    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.671612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.668129    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.671612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.670505    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.671612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.668996    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.672243 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.672226    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.672428 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.672400    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.673531 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.673513    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.673761 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.673724    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b616ef83-09fe-47a2-9d15-4ba5a0126998-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.674072 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.672759    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.690419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.690391    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvw8f\" (UniqueName: \"kubernetes.io/projected/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-kube-api-access-jvw8f\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.692054 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.692028    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.775885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.775759    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.775885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.775833    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.775885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.775858    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.776179 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.775920    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.776179 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.776121    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.776331 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.776305    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b616ef83-09fe-47a2-9d15-4ba5a0126998-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.776518 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.776499    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.776603 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.776336    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.776603 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.776543    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rlfk\" (UniqueName: \"kubernetes.io/projected/b616ef83-09fe-47a2-9d15-4ba5a0126998-kube-api-access-6rlfk\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.776706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.776648    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.776950 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.776928    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.777094 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.777004    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.777250 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.777228    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.777481 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.777461    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b616ef83-09fe-47a2-9d15-4ba5a0126998-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.778793 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.778767    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.779475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.779451    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.795910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.795875    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.796407 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.796381    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rlfk\" (UniqueName: \"kubernetes.io/projected/b616ef83-09fe-47a2-9d15-4ba5a0126998-kube-api-access-6rlfk\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:05.898919 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.898888    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:05.947831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:05.947585    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:06.229147 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:06.229111    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"]
Oct 29 22:31:06.229749 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:31:06.229715    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8d9cb1_a201_4c88_b3f2_72e5f2d4ec3f.slice/crio-34ddc56898a63d3f77dc2af4ee727ad6cab75831452916d3f709313699b0dbdf WatchSource:0}: Error finding container 34ddc56898a63d3f77dc2af4ee727ad6cab75831452916d3f709313699b0dbdf: Status 404 returned error can't find the container with id 34ddc56898a63d3f77dc2af4ee727ad6cab75831452916d3f709313699b0dbdf
Oct 29 22:31:06.232356 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:06.232320    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:31:06.232484 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:06.232399    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:31:06.232484 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:06.232448    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:31:06.290393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:06.290369    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"]
Oct 29 22:31:06.292231 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:31:06.292162    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb616ef83_09fe_47a2_9d15_4ba5a0126998.slice/crio-ba5501565a043825784151731b175b244f2a96bff1a4696eb8355ff59bbedf2e WatchSource:0}: Error finding container ba5501565a043825784151731b175b244f2a96bff1a4696eb8355ff59bbedf2e: Status 404 returned error can't find the container with id ba5501565a043825784151731b175b244f2a96bff1a4696eb8355ff59bbedf2e
Oct 29 22:31:06.295173 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:06.295128    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:31:06.295265 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:06.295203    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:31:06.295265 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:06.295236    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:31:06.310547 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:06.310510    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" event={"ID":"b616ef83-09fe-47a2-9d15-4ba5a0126998","Type":"ContainerStarted","Data":"ba5501565a043825784151731b175b244f2a96bff1a4696eb8355ff59bbedf2e"}
Oct 29 22:31:06.312539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:06.312505    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" event={"ID":"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f","Type":"ContainerStarted","Data":"34ddc56898a63d3f77dc2af4ee727ad6cab75831452916d3f709313699b0dbdf"}
Oct 29 22:31:07.407833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:07.407210    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" event={"ID":"b616ef83-09fe-47a2-9d15-4ba5a0126998","Type":"ContainerStarted","Data":"737a9b1fccf16c1391dd2c2ab48bee5f21be91a8ebdcb68c514b70bc7a3e5eb7"}
Oct 29 22:31:07.438838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:07.434391    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" event={"ID":"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f","Type":"ContainerStarted","Data":"a54f5da2d7d79612848efedce09b44c55a6546c0882ba30d88b10fcc9bee35c2"}
Oct 29 22:31:07.558834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:07.558101    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" podStartSLOduration=2.558083114 podStartE2EDuration="2.558083114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:07.557668275 +0000 UTC m=+1006.469111830" watchObservedRunningTime="2026-10-29 22:31:07.558083114 +0000 UTC m=+1006.469526668"
Oct 29 22:31:07.564882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:07.559919    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" podStartSLOduration=2.5598881970000003 podStartE2EDuration="2.559888197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:07.496575386 +0000 UTC m=+1006.408018941" watchObservedRunningTime="2026-10-29 22:31:07.559888197 +0000 UTC m=+1006.471331755"
Oct 29 22:31:07.899944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:07.899908    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:07.900955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:07.900931    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:31:07.901014 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:07.900985    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" podUID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 22:31:07.953834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:07.952661    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:07.953834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:07.952858    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:31:07.953834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:07.952908    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" podUID="b616ef83-09fe-47a2-9d15-4ba5a0126998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 29 22:31:08.900131 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:08.900093    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:31:08.900550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:08.900150    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" podUID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 22:31:08.958836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:08.958698    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:31:08.958836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:08.958761    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" podUID="b616ef83-09fe-47a2-9d15-4ba5a0126998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 29 22:31:09.899663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:09.899626    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:31:09.899948 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:09.899909    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" podUID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 22:31:09.949276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:09.949238    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:31:09.949875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:09.949843    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" podUID="b616ef83-09fe-47a2-9d15-4ba5a0126998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 29 22:31:10.899597 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:10.899408    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:31:10.899597 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:10.899501    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" podUID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 22:31:10.949030 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:10.948993    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:31:10.949319 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:10.949286    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" podUID="b616ef83-09fe-47a2-9d15-4ba5a0126998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 29 22:31:11.406707 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:11.406664    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:31:11.899331 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:11.899279    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:31:11.899569 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:11.899349    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" podUID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 22:31:11.953033 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:11.952994    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:31:11.953648 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:11.953617    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" podUID="b616ef83-09fe-47a2-9d15-4ba5a0126998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 29 22:31:12.907522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:12.907476    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:31:12.930836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:12.907783    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" podUID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 22:31:13.018838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:13.009615    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:31:13.018838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:13.009673    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" podUID="b616ef83-09fe-47a2-9d15-4ba5a0126998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 29 22:31:13.780610 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:13.779989    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:13.903837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:13.900428    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:31:13.903837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:13.900509    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" podUID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 22:31:13.951945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:13.948842    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:31:13.951945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:13.948903    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" podUID="b616ef83-09fe-47a2-9d15-4ba5a0126998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 29 22:31:14.032876 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:31:14.032744    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:31:14.904282 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:14.899777    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:31:14.904754 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:14.904702    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" podUID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 22:31:14.948034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:14.947976    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:14.949881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:14.949848    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:31:14.975339 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:14.975280    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" podUID="b616ef83-09fe-47a2-9d15-4ba5a0126998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 29 22:31:15.409836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:15.408932    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:15.677260 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:15.677160    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:31:15.736999 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:15.734986    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:15.736999 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:15.735068    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t" podUID="62ebc209-264e-49db-87fd-ca9c789386b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:15.736999 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:15.735173    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:15.806728 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:15.805003    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:15.806728 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:15.805070    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv" podUID="4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:15.964276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:15.948501    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:31:15.964276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:15.948585    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" podUID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 22:31:15.964276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:15.948958    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:15.990842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:15.949538    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:31:15.990842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:15.981989    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 22:31:16.030425 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:16.030051    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:16.030879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:16.030770    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" podUID="b616ef83-09fe-47a2-9d15-4ba5a0126998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 29 22:31:16.048740 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:16.031256    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:31:16.443377 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:16.424674    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:16.466533 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:16.466504    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:31:16.667009 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:16.666901    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:16.736041 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:16.702750    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:16.736041 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:16.703192    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:31:16.745431 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:16.743496    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:31:16.745431 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:16.743882    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:31:16.803140 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:16.803104    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:16.803418 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:16.803387    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj" podUID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:16.994901 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:16.994763    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:16.995327 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:16.995156    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl" podUID="220a1952-3291-485e-b6d4-4336b1cd1db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:17.006244 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:17.006107    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:17.013954 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:17.013925    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:31:17.030473 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:17.030420    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:17.048273 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:17.048238    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:31:17.078839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:17.057859    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:17.323240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:17.323120    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded"
Oct 29 22:31:17.468036 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:17.467995    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:31:17.565211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:17.565164    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:17.589118 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:17.589007    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82" podUID="6a3bfdf3-5fca-44b9-88d7-bf525139ee93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:17.703392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:17.703349    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:31:17.756982 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:17.756914    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:17.930297 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:17.899584    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:31:18.141070 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:17.964209    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:18.141362 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:18.141328    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" podUID="b616ef83-09fe-47a2-9d15-4ba5a0126998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:18.153273 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:18.008903    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:18.153273 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:18.150461    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" podUID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:18.153273 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:17.990459    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:31:18.153273 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:18.150855    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:31:18.153273 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:18.150900    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:31:18.154774 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:18.153828    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:31:18.154774 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:18.045928    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:18.175571 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:18.175543    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:31:19.151600 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:19.151549    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:19.152190 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:19.151628    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" podUID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:19.183025 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:19.182971    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:19.183246 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:19.183059    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" podUID="b616ef83-09fe-47a2-9d15-4ba5a0126998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:20.085838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:20.085621    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:20.086131 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:20.086108    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:20.087547 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:20.087478    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:31:20.195389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:20.195198    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:20.195389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:20.195267    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" podUID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:20.294421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:20.294393    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:20.294715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:20.294695    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:20.295640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:20.295617    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:31:21.087426 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.087392    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"]
Oct 29 22:31:21.093395 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.093368    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.123213 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.123170    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"]
Oct 29 22:31:21.139759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.139724    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.139986 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.139795    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.140077 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.140053    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.140138 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.140096    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.140216 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.140199    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnmbn\" (UniqueName: \"kubernetes.io/projected/6167de0d-b93b-4c88-8239-2c697bbd7c7a-kube-api-access-tnmbn\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.140277 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.140261    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.140345 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.140329    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.140440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.140411    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.140487 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.140459    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.241976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.241945    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.242457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.241985    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.242457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.242036    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnmbn\" (UniqueName: \"kubernetes.io/projected/6167de0d-b93b-4c88-8239-2c697bbd7c7a-kube-api-access-tnmbn\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.242457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.242187    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.242457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.242241    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.242457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.242272    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.242457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.242296    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.242457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.242326    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.242457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.242395    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.242973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.242488    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.242973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.242503    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.242973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.242575    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.242973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.242671    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.243352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.243342    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.244946 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.244919    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.245354 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.245336    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.263490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.263454    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnmbn\" (UniqueName: \"kubernetes.io/projected/6167de0d-b93b-4c88-8239-2c697bbd7c7a-kube-api-access-tnmbn\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.263666 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.263540    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.410614 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.410560    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:21.747129 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.747096    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"]
Oct 29 22:31:21.749248 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:31:21.749222    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6167de0d_b93b_4c88_8239_2c697bbd7c7a.slice/crio-c85c9759c283b9917c05ba26c729bc6740eaae9fa3339a7f841dd869ea9b0b63 WatchSource:0}: Error finding container c85c9759c283b9917c05ba26c729bc6740eaae9fa3339a7f841dd869ea9b0b63: Status 404 returned error can't find the container with id c85c9759c283b9917c05ba26c729bc6740eaae9fa3339a7f841dd869ea9b0b63
Oct 29 22:31:21.751844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.751781    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:31:21.751933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.751874    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:31:21.751973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:21.751939    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:31:22.441823 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:22.441763    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt" event={"ID":"6167de0d-b93b-4c88-8239-2c697bbd7c7a","Type":"ContainerStarted","Data":"64705d378f0851ba2b019888b290b03fff15dda939e5290e62c89b123e0c75f0"}
Oct 29 22:31:22.442337 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:22.441859    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt" event={"ID":"6167de0d-b93b-4c88-8239-2c697bbd7c7a","Type":"ContainerStarted","Data":"c85c9759c283b9917c05ba26c729bc6740eaae9fa3339a7f841dd869ea9b0b63"}
Oct 29 22:31:22.513328 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:22.513268    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt" podStartSLOduration=1.51325227 podStartE2EDuration="1.51325227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:22.497958543 +0000 UTC m=+1021.409402139" watchObservedRunningTime="2026-10-29 22:31:22.51325227 +0000 UTC m=+1021.424695823"
Oct 29 22:31:23.413959 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:23.413924    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:23.414389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:23.414361    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 29 22:31:23.414561 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:23.414538    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt" podUID="6167de0d-b93b-4c88-8239-2c697bbd7c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 29 22:31:24.411278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:24.411244    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 29 22:31:24.411772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:24.411307    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt" podUID="6167de0d-b93b-4c88-8239-2c697bbd7c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 29 22:31:25.411379 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:25.411345    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 29 22:31:25.411826 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:25.411398    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt" podUID="6167de0d-b93b-4c88-8239-2c697bbd7c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 29 22:31:26.441989 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:26.441949    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 29 22:31:26.442620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:26.442583    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt" podUID="6167de0d-b93b-4c88-8239-2c697bbd7c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 29 22:31:27.174694 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:27.173953    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:27.412076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:27.412036    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 29 22:31:27.412358 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:27.412328    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt" podUID="6167de0d-b93b-4c88-8239-2c697bbd7c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 29 22:31:27.708093 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:27.708049    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:27.904837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:27.903961    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.001900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:27.997971    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.175027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:28.174962    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.176366 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:28.176345    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:31:28.436837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:28.436316    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 29 22:31:28.436837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:28.436371    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt" podUID="6167de0d-b93b-4c88-8239-2c697bbd7c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 29 22:31:28.459852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:28.447436    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.472513 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:28.472065    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.746919 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:28.746794    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.748419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:28.748395    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:31:28.998909 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:28.991235    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 22:31:28.998909 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:28.991687    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:31:29.004845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:28.999253    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:31:29.004845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:28.999712    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:31:29.124847 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:29.118086    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:29.302951 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:29.292092    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:29.414404 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:29.414365    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 29 22:31:29.414708 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:29.414680    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt" podUID="6167de0d-b93b-4c88-8239-2c697bbd7c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 29 22:31:29.833845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:29.832952    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:31:30.093907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:29.996728    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:30.119664 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:30.119396    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:31:30.219834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:30.212954    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:31:30.295350 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:30.295288    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:30.345220 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:30.345112    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:31:30.459158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:30.459111    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 29 22:31:30.698770 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:30.698730    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt" podUID="6167de0d-b93b-4c88-8239-2c697bbd7c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 29 22:31:30.699931 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:30.459843    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Oct 29 22:31:31.011418 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.011315    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:31.124863 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.013291    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:31.125174 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.094931    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:31:31.127091 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.096970    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:31:31.136905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.121041    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:31.160289 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.160250    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:31.160560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.160535    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t" podUID="62ebc209-264e-49db-87fd-ca9c789386b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:31.169513 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.169481    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:31:31.171547 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.171524    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:31:31.187934 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.187902    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:31:31.218220 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.218174    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:31.245304 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.245269    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:31.245576 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.245547    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv" podUID="4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:31.370896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.362585    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:31.471099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.469656    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 29 22:31:31.471099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.469718    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt" podUID="6167de0d-b93b-4c88-8239-2c697bbd7c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 29 22:31:31.553236 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.552536    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:31:31.560755 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:31.559601    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:31:32.411244 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:32.411192    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 29 22:31:32.411851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:32.411269    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt" podUID="6167de0d-b93b-4c88-8239-2c697bbd7c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 29 22:31:33.416330 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:33.416299    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:33.416756 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:33.416729    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:33.417405 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:33.417385    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:31:36.142121 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.142086    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"]
Oct 29 22:31:36.147928 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.147900    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.195049 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.195015    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"]
Oct 29 22:31:36.294538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.294488    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.294756 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.294626    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707246aa-d09a-4f11-96f3-2778aa943e1b-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.294756 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.294689    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.294756 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.294724    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.295001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.294981    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.295063 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.295026    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.295063 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.295052    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.295211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.295187    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt2nk\" (UniqueName: \"kubernetes.io/projected/707246aa-d09a-4f11-96f3-2778aa943e1b-kube-api-access-bt2nk\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.295265 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.295250    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.396782 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.396679    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.396782 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.396717    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.396782 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.396752    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt2nk\" (UniqueName: \"kubernetes.io/projected/707246aa-d09a-4f11-96f3-2778aa943e1b-kube-api-access-bt2nk\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.397081 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.396784    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.397081 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.396905    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.397081 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.396952    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707246aa-d09a-4f11-96f3-2778aa943e1b-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.397081 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.396985    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.397081 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.397022    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.397341 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.397098    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.397341 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.397210    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.397341 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.397248    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.397502 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.397344    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.397502 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.397445    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.397925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.397906    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707246aa-d09a-4f11-96f3-2778aa943e1b-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.399279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.399255    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.399706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.399686    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.417188 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.417151    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.417325 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.417227    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt2nk\" (UniqueName: \"kubernetes.io/projected/707246aa-d09a-4f11-96f3-2778aa943e1b-kube-api-access-bt2nk\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.473041 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.473001    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:36.804441 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.804341    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"]
Oct 29 22:31:36.804441 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:31:36.804427    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707246aa_d09a_4f11_96f3_2778aa943e1b.slice/crio-956db9e631d786f605d2277400e8e14365c817c75b898a7d6083aa39db689d49 WatchSource:0}: Error finding container 956db9e631d786f605d2277400e8e14365c817c75b898a7d6083aa39db689d49: Status 404 returned error can't find the container with id 956db9e631d786f605d2277400e8e14365c817c75b898a7d6083aa39db689d49
Oct 29 22:31:36.807299 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.807261    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:31:36.807400 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.807331    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:31:36.807400 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:36.807361    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:31:37.151425 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:37.151387    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" event={"ID":"707246aa-d09a-4f11-96f3-2778aa943e1b","Type":"ContainerStarted","Data":"3acdf10efab044392e57975c53cce62be10ca01d7db448cb636877d281cfbaf3"}
Oct 29 22:31:37.151942 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:37.151482    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" event={"ID":"707246aa-d09a-4f11-96f3-2778aa943e1b","Type":"ContainerStarted","Data":"956db9e631d786f605d2277400e8e14365c817c75b898a7d6083aa39db689d49"}
Oct 29 22:31:37.209837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:37.206049    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podStartSLOduration=1.206028331 podStartE2EDuration="1.206028331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:37.20218156 +0000 UTC m=+1036.113625115" watchObservedRunningTime="2026-10-29 22:31:37.206028331 +0000 UTC m=+1036.117471885"
Oct 29 22:31:37.473466 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:37.473360    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:37.475079 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:37.475047    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:31:37.475228 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:37.475123    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 29 22:31:38.474385 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:38.474352    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:31:38.474801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:38.474426    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 29 22:31:39.473455 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:39.473410    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:31:39.473648 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:39.473474    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 29 22:31:40.474095 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:40.474056    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:31:40.474491 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:40.474134    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 29 22:31:41.474103 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:41.474063    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:31:41.474617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:41.474138    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 29 22:31:42.477396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:42.477356    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:31:42.478042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:42.477974    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 29 22:31:42.615111 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:42.615072    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:31:43.476341 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:43.476302    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:31:43.476685 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:43.476645    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 29 22:31:43.916094 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:31:43.912077    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:31:44.479758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:44.479285    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:31:44.479758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:44.479361    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 29 22:31:45.476072 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:45.476032    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:31:45.476769 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:45.476732    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 29 22:31:46.185836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:46.181161    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.590112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:46.589979    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.681220 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:46.681097    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.775849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:46.775288    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:46.775849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:46.775371    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj" podUID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.062796 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:47.062745    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:47.063249 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:47.063206    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl" podUID="220a1952-3291-485e-b6d4-4336b1cd1db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.162954 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:47.149045    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:31:47.202529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:47.186306    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.202529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:47.187763    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:31:47.202529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:47.201137    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:31:47.291019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:47.290977    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.487955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:47.469163    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:47.487955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:47.479638    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82" podUID="6a3bfdf3-5fca-44b9-88d7-bf525139ee93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.518114 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:47.518064    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.573309 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:47.572914    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:47.573309 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:47.573001    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.607879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:47.605244    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:31:47.923902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:47.772222    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:31:48.187044 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:48.186927    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:48.210843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:48.191291    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:48.218632 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:48.198327    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:31:48.243751 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:48.218974    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:31:48.245038 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:48.200263    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:31:48.394838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:48.369141    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:48.532900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:48.531910    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:48.532900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:48.532347    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:31:48.619040 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:48.609543    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:48.625969 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:48.625928    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:48.664372 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:48.664320    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:49.168839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:49.166760    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:49.168839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:49.166859    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67" podUID="52d2294b-e879-4782-b9e9-526f2aeecdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:49.228846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:49.223975    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:49.228846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:49.224367    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:49.257836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:49.256146    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Oct 29 22:31:49.278971 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:49.278919    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:49.402698 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:49.402644    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:49.406306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:49.406272    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:31:49.482039 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:49.481943    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:49.482297 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:49.482266    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt" podUID="6167de0d-b93b-4c88-8239-2c697bbd7c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:49.536945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:49.536894    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:49.636959 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:49.636909    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:49.701022 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:49.700971    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:49.701342 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:49.701294    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:50.239945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:50.239899    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:31:50.245216 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:50.240205    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:50.297837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:50.294287    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:50.420838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:50.417452    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:50.437307 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:50.437265    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:50.505522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:50.505434    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:50.505785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:50.505742    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:50.609146 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:50.609097    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:31:50.648010 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:50.647949    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:50.708667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:50.656709    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Oct 29 22:31:50.769517 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:50.768954    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:50.769517 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:50.769020    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:51.305384 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:51.305329    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:51.332681 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:51.332490    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:51.553432 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:51.541058    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:31:51.614941 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:51.614720    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:51.618870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:51.618767    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:51.633258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:51.633226    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:31:51.633953 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:51.633934    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:31:51.635037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:51.635018    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:51.635230 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:51.635215    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:31:51.635382 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:51.635368    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:31:51.639539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:51.639507    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:31:51.652609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:51.652584    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:31:51.810426 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:51.810344    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:51.811236 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:51.810711    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:52.242758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:52.242722    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:52.643958 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:52.643933    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:31:58.501834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:58.500867    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"]
Oct 29 22:31:58.507884 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:58.507338    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"
Oct 29 22:31:58.522047 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:58.516582    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"]
Oct 29 22:31:58.528626 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:58.528592    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:31:58.528922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:58.528907    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:31:58.606448 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:58.606409    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4mpq\" (UniqueName: \"kubernetes.io/projected/53fdfe33-36d4-493a-ac2b-520cdd5977fa-kube-api-access-t4mpq\") pod \"maas-api-revoke-sub-a6c95b37-vlgjr\" (UID: \"53fdfe33-36d4-493a-ac2b-520cdd5977fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"
Oct 29 22:31:58.606659 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:58.606587    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53fdfe33-36d4-493a-ac2b-520cdd5977fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c95b37-vlgjr\" (UID: \"53fdfe33-36d4-493a-ac2b-520cdd5977fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"
Oct 29 22:31:58.606826 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:58.606796    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53fdfe33-36d4-493a-ac2b-520cdd5977fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c95b37-vlgjr\" (UID: \"53fdfe33-36d4-493a-ac2b-520cdd5977fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"
Oct 29 22:31:58.707873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:58.707792    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4mpq\" (UniqueName: \"kubernetes.io/projected/53fdfe33-36d4-493a-ac2b-520cdd5977fa-kube-api-access-t4mpq\") pod \"maas-api-revoke-sub-a6c95b37-vlgjr\" (UID: \"53fdfe33-36d4-493a-ac2b-520cdd5977fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"
Oct 29 22:31:58.708060 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:58.707941    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53fdfe33-36d4-493a-ac2b-520cdd5977fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c95b37-vlgjr\" (UID: \"53fdfe33-36d4-493a-ac2b-520cdd5977fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"
Oct 29 22:31:58.708108 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:58.708091    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53fdfe33-36d4-493a-ac2b-520cdd5977fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c95b37-vlgjr\" (UID: \"53fdfe33-36d4-493a-ac2b-520cdd5977fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"
Oct 29 22:31:58.708654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:58.708625    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53fdfe33-36d4-493a-ac2b-520cdd5977fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c95b37-vlgjr\" (UID: \"53fdfe33-36d4-493a-ac2b-520cdd5977fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"
Oct 29 22:31:58.710537 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:58.710515    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53fdfe33-36d4-493a-ac2b-520cdd5977fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c95b37-vlgjr\" (UID: \"53fdfe33-36d4-493a-ac2b-520cdd5977fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"
Oct 29 22:31:58.728736 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:58.728702    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4mpq\" (UniqueName: \"kubernetes.io/projected/53fdfe33-36d4-493a-ac2b-520cdd5977fa-kube-api-access-t4mpq\") pod \"maas-api-revoke-sub-a6c95b37-vlgjr\" (UID: \"53fdfe33-36d4-493a-ac2b-520cdd5977fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"
Oct 29 22:31:58.840551 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:58.840459    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"
Oct 29 22:31:59.146652 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:59.146618    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"]
Oct 29 22:31:59.147101 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:31:59.147076    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fdfe33_36d4_493a_ac2b_520cdd5977fa.slice/crio-78b5f22e5be98157ea85b8b52d1bf1fadaeb323004a26f5668fdfa8e2af990d0 WatchSource:0}: Error finding container 78b5f22e5be98157ea85b8b52d1bf1fadaeb323004a26f5668fdfa8e2af990d0: Status 404 returned error can't find the container with id 78b5f22e5be98157ea85b8b52d1bf1fadaeb323004a26f5668fdfa8e2af990d0
Oct 29 22:31:59.702789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:59.702755    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr" event={"ID":"53fdfe33-36d4-493a-ac2b-520cdd5977fa","Type":"ContainerStarted","Data":"c57bc7d4bec46ad2b3c78557e276ec076647c1cf7e9ac89cd64c053c21afae08"}
Oct 29 22:31:59.703224 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:59.702805    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr" event={"ID":"53fdfe33-36d4-493a-ac2b-520cdd5977fa","Type":"ContainerStarted","Data":"78b5f22e5be98157ea85b8b52d1bf1fadaeb323004a26f5668fdfa8e2af990d0"}
Oct 29 22:31:59.745996 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:31:59.745756    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr" podStartSLOduration=1.745736943 podStartE2EDuration="1.745736943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:59.742545989 +0000 UTC m=+1058.653989542" watchObservedRunningTime="2026-10-29 22:31:59.745736943 +0000 UTC m=+1058.657180496"
Oct 29 22:32:00.600365 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:00.600325    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"]
Oct 29 22:32:00.605195 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:00.605169    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"
Oct 29 22:32:00.628676 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:00.628644    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"]
Oct 29 22:32:00.638258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:00.638226    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d23143c1-8b6a-4975-a219-832af8f64116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1696adc8-5vf6w\" (UID: \"d23143c1-8b6a-4975-a219-832af8f64116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"
Oct 29 22:32:00.638451 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:00.638323    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs7dw\" (UniqueName: \"kubernetes.io/projected/d23143c1-8b6a-4975-a219-832af8f64116-kube-api-access-xs7dw\") pod \"maas-api-revoke-sub-1696adc8-5vf6w\" (UID: \"d23143c1-8b6a-4975-a219-832af8f64116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"
Oct 29 22:32:00.638451 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:00.638445    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d23143c1-8b6a-4975-a219-832af8f64116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1696adc8-5vf6w\" (UID: \"d23143c1-8b6a-4975-a219-832af8f64116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"
Oct 29 22:32:00.712750 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:00.712707    4559 generic.go:356] "Generic (PLEG): container finished" podID="53fdfe33-36d4-493a-ac2b-520cdd5977fa" containerID="c57bc7d4bec46ad2b3c78557e276ec076647c1cf7e9ac89cd64c053c21afae08" exitCode=0
Oct 29 22:32:00.713245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:00.712767    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr" event={"ID":"53fdfe33-36d4-493a-ac2b-520cdd5977fa","Type":"ContainerDied","Data":"c57bc7d4bec46ad2b3c78557e276ec076647c1cf7e9ac89cd64c053c21afae08"}
Oct 29 22:32:00.740357 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:00.740312    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d23143c1-8b6a-4975-a219-832af8f64116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1696adc8-5vf6w\" (UID: \"d23143c1-8b6a-4975-a219-832af8f64116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"
Oct 29 22:32:00.740570 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:00.740419    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs7dw\" (UniqueName: \"kubernetes.io/projected/d23143c1-8b6a-4975-a219-832af8f64116-kube-api-access-xs7dw\") pod \"maas-api-revoke-sub-1696adc8-5vf6w\" (UID: \"d23143c1-8b6a-4975-a219-832af8f64116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"
Oct 29 22:32:00.740570 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:00.740532    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d23143c1-8b6a-4975-a219-832af8f64116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1696adc8-5vf6w\" (UID: \"d23143c1-8b6a-4975-a219-832af8f64116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"
Oct 29 22:32:00.741176 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:00.741155    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d23143c1-8b6a-4975-a219-832af8f64116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1696adc8-5vf6w\" (UID: \"d23143c1-8b6a-4975-a219-832af8f64116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"
Oct 29 22:32:00.743164 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:00.743139    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d23143c1-8b6a-4975-a219-832af8f64116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1696adc8-5vf6w\" (UID: \"d23143c1-8b6a-4975-a219-832af8f64116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"
Oct 29 22:32:00.762998 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:00.762964    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs7dw\" (UniqueName: \"kubernetes.io/projected/d23143c1-8b6a-4975-a219-832af8f64116-kube-api-access-xs7dw\") pod \"maas-api-revoke-sub-1696adc8-5vf6w\" (UID: \"d23143c1-8b6a-4975-a219-832af8f64116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"
Oct 29 22:32:00.919550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:00.919509    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"
Oct 29 22:32:01.244710 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:01.244684    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"]
Oct 29 22:32:01.722308 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:01.722266    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w" event={"ID":"d23143c1-8b6a-4975-a219-832af8f64116","Type":"ContainerStarted","Data":"9a7a64e09170525f0eb3e1b109dc5f04a955f6f7c82d9c6c9d4117e51a3fa95e"}
Oct 29 22:32:01.722836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:01.722339    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w" event={"ID":"d23143c1-8b6a-4975-a219-832af8f64116","Type":"ContainerStarted","Data":"025ad65c556b7bcb86595f8c67a768ddf83fa130e8dad7169090210ace403059"}
Oct 29 22:32:01.866344 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:01.866229    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w" podStartSLOduration=1.866210346 podStartE2EDuration="1.866210346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:01.786789446 +0000 UTC m=+1060.698233014" watchObservedRunningTime="2026-10-29 22:32:01.866210346 +0000 UTC m=+1060.777653902"
Oct 29 22:32:02.032543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:02.032517    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"
Oct 29 22:32:02.132011 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:02.131917    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53fdfe33-36d4-493a-ac2b-520cdd5977fa-kube-api-access-t4mpq\" (UniqueName: \"kubernetes.io/projected/53fdfe33-36d4-493a-ac2b-520cdd5977fa-kube-api-access-t4mpq\") pod \"53fdfe33-36d4-493a-ac2b-520cdd5977fa\" (UID: \"53fdfe33-36d4-493a-ac2b-520cdd5977fa\") "
Oct 29 22:32:02.132011 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:02.131965    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53fdfe33-36d4-493a-ac2b-520cdd5977fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53fdfe33-36d4-493a-ac2b-520cdd5977fa-maas-api-openshift-ca\") pod \"53fdfe33-36d4-493a-ac2b-520cdd5977fa\" (UID: \"53fdfe33-36d4-493a-ac2b-520cdd5977fa\") "
Oct 29 22:32:02.132233 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:02.132214    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53fdfe33-36d4-493a-ac2b-520cdd5977fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53fdfe33-36d4-493a-ac2b-520cdd5977fa-maas-api-xks-ca\") pod \"53fdfe33-36d4-493a-ac2b-520cdd5977fa\" (UID: \"53fdfe33-36d4-493a-ac2b-520cdd5977fa\") "
Oct 29 22:32:02.132354 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:02.132327    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53fdfe33-36d4-493a-ac2b-520cdd5977fa-maas-api-openshift-ca" pod "53fdfe33-36d4-493a-ac2b-520cdd5977fa" (UID: "53fdfe33-36d4-493a-ac2b-520cdd5977fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:02.132655 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:02.132635    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53fdfe33-36d4-493a-ac2b-520cdd5977fa-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:02.134322 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:02.134294    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53fdfe33-36d4-493a-ac2b-520cdd5977fa-maas-api-xks-ca" pod "53fdfe33-36d4-493a-ac2b-520cdd5977fa" (UID: "53fdfe33-36d4-493a-ac2b-520cdd5977fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:02.134639 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:02.134614    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fdfe33-36d4-493a-ac2b-520cdd5977fa-kube-api-access-t4mpq" pod "53fdfe33-36d4-493a-ac2b-520cdd5977fa" (UID: "53fdfe33-36d4-493a-ac2b-520cdd5977fa"). InnerVolumeSpecName "kube-api-access-t4mpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:02.233646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:02.233613    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53fdfe33-36d4-493a-ac2b-520cdd5977fa-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:02.233646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:02.233644    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4mpq\" (UniqueName: \"kubernetes.io/projected/53fdfe33-36d4-493a-ac2b-520cdd5977fa-kube-api-access-t4mpq\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:02.732562 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:02.732526    4559 generic.go:356] "Generic (PLEG): container finished" podID="d23143c1-8b6a-4975-a219-832af8f64116" containerID="9a7a64e09170525f0eb3e1b109dc5f04a955f6f7c82d9c6c9d4117e51a3fa95e" exitCode=0
Oct 29 22:32:02.733170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:02.732627    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w" event={"ID":"d23143c1-8b6a-4975-a219-832af8f64116","Type":"ContainerDied","Data":"9a7a64e09170525f0eb3e1b109dc5f04a955f6f7c82d9c6c9d4117e51a3fa95e"}
Oct 29 22:32:02.734733 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:02.734707    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"
Oct 29 22:32:02.734923 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:02.734699    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr" event={"ID":"53fdfe33-36d4-493a-ac2b-520cdd5977fa","Type":"ContainerDied","Data":"78b5f22e5be98157ea85b8b52d1bf1fadaeb323004a26f5668fdfa8e2af990d0"}
Oct 29 22:32:02.734923 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:02.734886    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78b5f22e5be98157ea85b8b52d1bf1fadaeb323004a26f5668fdfa8e2af990d0"
Oct 29 22:32:03.921371 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:03.921339    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"
Oct 29 22:32:03.967400 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:03.967360    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d23143c1-8b6a-4975-a219-832af8f64116-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d23143c1-8b6a-4975-a219-832af8f64116-maas-api-xks-ca\") pod \"d23143c1-8b6a-4975-a219-832af8f64116\" (UID: \"d23143c1-8b6a-4975-a219-832af8f64116\") "
Oct 29 22:32:03.967620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:03.967579    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d23143c1-8b6a-4975-a219-832af8f64116-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d23143c1-8b6a-4975-a219-832af8f64116-maas-api-openshift-ca\") pod \"d23143c1-8b6a-4975-a219-832af8f64116\" (UID: \"d23143c1-8b6a-4975-a219-832af8f64116\") "
Oct 29 22:32:03.967724 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:03.967706    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d23143c1-8b6a-4975-a219-832af8f64116-kube-api-access-xs7dw\" (UniqueName: \"kubernetes.io/projected/d23143c1-8b6a-4975-a219-832af8f64116-kube-api-access-xs7dw\") pod \"d23143c1-8b6a-4975-a219-832af8f64116\" (UID: \"d23143c1-8b6a-4975-a219-832af8f64116\") "
Oct 29 22:32:03.968004 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:03.967972    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d23143c1-8b6a-4975-a219-832af8f64116-maas-api-openshift-ca" pod "d23143c1-8b6a-4975-a219-832af8f64116" (UID: "d23143c1-8b6a-4975-a219-832af8f64116"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:03.968332 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:03.968311    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d23143c1-8b6a-4975-a219-832af8f64116-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:03.970008 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:03.969976    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23143c1-8b6a-4975-a219-832af8f64116-maas-api-xks-ca" pod "d23143c1-8b6a-4975-a219-832af8f64116" (UID: "d23143c1-8b6a-4975-a219-832af8f64116"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:03.970624 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:03.970598    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23143c1-8b6a-4975-a219-832af8f64116-kube-api-access-xs7dw" pod "d23143c1-8b6a-4975-a219-832af8f64116" (UID: "d23143c1-8b6a-4975-a219-832af8f64116"). InnerVolumeSpecName "kube-api-access-xs7dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:04.070282 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:04.070190    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs7dw\" (UniqueName: \"kubernetes.io/projected/d23143c1-8b6a-4975-a219-832af8f64116-kube-api-access-xs7dw\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:04.070282 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:04.070222    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d23143c1-8b6a-4975-a219-832af8f64116-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:04.750566 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:04.750526    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w" event={"ID":"d23143c1-8b6a-4975-a219-832af8f64116","Type":"ContainerDied","Data":"025ad65c556b7bcb86595f8c67a768ddf83fa130e8dad7169090210ace403059"}
Oct 29 22:32:04.750566 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:04.750565    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"
Oct 29 22:32:04.750825 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:04.750591    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="025ad65c556b7bcb86595f8c67a768ddf83fa130e8dad7169090210ace403059"
Oct 29 22:32:10.282836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:10.281975    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 22:32:10.299102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:10.288466    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.306554 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:10.306469    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:32:10.434973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:10.434927    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.545062 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:10.544959    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.620845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:10.620153    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.622382 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:10.621576    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.667966 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:10.667914    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:11.287225 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:11.286710    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:11.288984 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:11.288885    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:32:11.311960 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:11.311911    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:32:11.476839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:11.462234    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:32:11.477235 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:11.477197    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:32:11.484949 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:11.484896    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:32:11.585905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:11.545703    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:11.607836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:11.589529    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:32:11.947330 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:11.783987    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:11.951978 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:11.799455    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:32:12.029863 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:12.029837    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:32:12.062329 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:12.062303    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:32:12.358984 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:12.358882    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:12.564838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:12.391938    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 22:32:12.564838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:12.392000    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:32:12.589334 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:12.589292    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:32:12.956845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:12.952935    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:32:12.967170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:12.967130    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.319897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:13.315903    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.556166 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:13.556119    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:32:13.603898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:13.564886    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 22:32:13.603898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:13.564968    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.617028 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:13.616970    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:32:14.056060 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:14.055995    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:32:14.343889 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:32:14.285637    4559 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.037s"
Oct 29 22:32:14.344296 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:14.344270    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"]
Oct 29 22:32:14.348842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:14.344644    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:32:14.420590 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:14.359988    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7" podUID="b15b2407-227c-40f8-95a9-4bfad483a2bb" containerName="istio-proxy" containerID="cri-o://2b0b55419a6269f1996fd04025fef6fff0f2f58852f95c980771fa74ea20f055" gracePeriod=30
Oct 29 22:32:14.477385 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:14.477348    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:32:14.500188 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:32:14.500133    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:32:14.613574 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:14.613470    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:14.614619 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:14.614588    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:32:14.646869 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:14.634572    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:14.740540 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:14.740505    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:32:14.788972 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:14.788928    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:15.682835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:15.655091    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:32:15.682835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:15.658522    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:32:15.807839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:15.797885    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:15.893836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:15.892964    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"]
Oct 29 22:32:15.894034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:15.893956    4559 cpu_manager.go:414] "containerMap: removing container" podUID="53fdfe33-36d4-493a-ac2b-520cdd5977fa" containerName="revoke-keys"
Oct 29 22:32:15.894034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:15.893979    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53fdfe33-36d4-493a-ac2b-520cdd5977fa" containerName="revoke-keys"
Oct 29 22:32:15.894034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:15.893992    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d23143c1-8b6a-4975-a219-832af8f64116" containerName="revoke-keys"
Oct 29 22:32:15.894034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:15.894001    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d23143c1-8b6a-4975-a219-832af8f64116" containerName="revoke-keys"
Oct 29 22:32:15.894232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:15.894161    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="53fdfe33-36d4-493a-ac2b-520cdd5977fa" containerName="revoke-keys"
Oct 29 22:32:15.894232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:15.894173    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d23143c1-8b6a-4975-a219-832af8f64116" containerName="revoke-keys"
Oct 29 22:32:15.939895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:15.938462    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"]
Oct 29 22:32:15.939895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:15.938593    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"
Oct 29 22:32:15.964847 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:15.962583    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:32:15.983430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:15.966898    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:32:15.983430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:15.969153    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:32:15.983430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:15.975532    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:32:16.000711 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:15.999953    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:32:16.002319 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.000885    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:16.027897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.027869    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:32:16.142890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.142797    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d747a8f-sptx8\" (UID: \"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"
Oct 29 22:32:16.143090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.143014    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d747a8f-sptx8\" (UID: \"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"
Oct 29 22:32:16.143155 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.143109    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqcdq\" (UniqueName: \"kubernetes.io/projected/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-kube-api-access-wqcdq\") pod \"maas-api-revoke-sub-3d747a8f-sptx8\" (UID: \"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"
Oct 29 22:32:16.245090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.244997    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d747a8f-sptx8\" (UID: \"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"
Oct 29 22:32:16.245273 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.245089    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d747a8f-sptx8\" (UID: \"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"
Oct 29 22:32:16.245273 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.245146    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqcdq\" (UniqueName: \"kubernetes.io/projected/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-kube-api-access-wqcdq\") pod \"maas-api-revoke-sub-3d747a8f-sptx8\" (UID: \"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"
Oct 29 22:32:16.245648 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.245627    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d747a8f-sptx8\" (UID: \"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"
Oct 29 22:32:16.247881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.247853    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d747a8f-sptx8\" (UID: \"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"
Oct 29 22:32:16.281193 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.281159    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqcdq\" (UniqueName: \"kubernetes.io/projected/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-kube-api-access-wqcdq\") pod \"maas-api-revoke-sub-3d747a8f-sptx8\" (UID: \"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"
Oct 29 22:32:16.347771 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.347731    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"
Oct 29 22:32:16.486437 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.486397    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"]
Oct 29 22:32:16.491854 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.491825    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"
Oct 29 22:32:16.508856 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.508775    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"]
Oct 29 22:32:16.650963 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.650925    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l26f2\" (UniqueName: \"kubernetes.io/projected/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-kube-api-access-l26f2\") pod \"maas-api-revoke-sub-0a0ae17c-5xsdg\" (UID: \"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"
Oct 29 22:32:16.651148 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.651077    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0ae17c-5xsdg\" (UID: \"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"
Oct 29 22:32:16.651243 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.651192    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0ae17c-5xsdg\" (UID: \"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"
Oct 29 22:32:16.658776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.658742    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"]
Oct 29 22:32:16.658957 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:32:16.658848    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44b9797_70d5_4a9e_a5bb_9b6afe189ab0.slice/crio-5b8bba117107bba5bc1e7fff0c94f58c875464453b401987c56965519a5f9003 WatchSource:0}: Error finding container 5b8bba117107bba5bc1e7fff0c94f58c875464453b401987c56965519a5f9003: Status 404 returned error can't find the container with id 5b8bba117107bba5bc1e7fff0c94f58c875464453b401987c56965519a5f9003
Oct 29 22:32:16.752474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.752374    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0ae17c-5xsdg\" (UID: \"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"
Oct 29 22:32:16.753009 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.752472    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0ae17c-5xsdg\" (UID: \"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"
Oct 29 22:32:16.753009 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.752609    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l26f2\" (UniqueName: \"kubernetes.io/projected/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-kube-api-access-l26f2\") pod \"maas-api-revoke-sub-0a0ae17c-5xsdg\" (UID: \"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"
Oct 29 22:32:16.753258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.753240    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0ae17c-5xsdg\" (UID: \"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"
Oct 29 22:32:16.755083 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.755058    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0ae17c-5xsdg\" (UID: \"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"
Oct 29 22:32:16.772991 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.772957    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l26f2\" (UniqueName: \"kubernetes.io/projected/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-kube-api-access-l26f2\") pod \"maas-api-revoke-sub-0a0ae17c-5xsdg\" (UID: \"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"
Oct 29 22:32:16.810056 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.810013    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"
Oct 29 22:32:16.888904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:16.888866    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8" event={"ID":"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0","Type":"ContainerStarted","Data":"5b8bba117107bba5bc1e7fff0c94f58c875464453b401987c56965519a5f9003"}
Oct 29 22:32:17.119523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:17.119484    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"]
Oct 29 22:32:17.119642 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:32:17.119564    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbe0061_a4e6_4a4d_9be9_5b8a0b6c615f.slice/crio-b913cf23b3b36e6c96d452549b4fc82b04cae7ac74f520c9839eed0ad0379a3b WatchSource:0}: Error finding container b913cf23b3b36e6c96d452549b4fc82b04cae7ac74f520c9839eed0ad0379a3b: Status 404 returned error can't find the container with id b913cf23b3b36e6c96d452549b4fc82b04cae7ac74f520c9839eed0ad0379a3b
Oct 29 22:32:17.899069 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:17.898971    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg" event={"ID":"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f","Type":"ContainerStarted","Data":"020710e28adc289961941ac338a8cac45e4168136eea0d1aae2421f3e826392d"}
Oct 29 22:32:17.899545 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:17.899075    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg" event={"ID":"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f","Type":"ContainerStarted","Data":"b913cf23b3b36e6c96d452549b4fc82b04cae7ac74f520c9839eed0ad0379a3b"}
Oct 29 22:32:17.901683 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:17.901650    4559 generic.go:356] "Generic (PLEG): container finished" podID="b44b9797-70d5-4a9e-a5bb-9b6afe189ab0" containerID="5e8a305d44751437be93a4d590627ed2c4f131c0f488356af653d7724607065c" exitCode=0
Oct 29 22:32:17.901861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:17.901735    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8" event={"ID":"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0","Type":"ContainerDied","Data":"5e8a305d44751437be93a4d590627ed2c4f131c0f488356af653d7724607065c"}
Oct 29 22:32:17.941858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:17.941781    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg" podStartSLOduration=1.941767442 podStartE2EDuration="1.941767442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:17.938521294 +0000 UTC m=+1076.849964849" watchObservedRunningTime="2026-10-29 22:32:17.941767442 +0000 UTC m=+1076.853210996"
Oct 29 22:32:18.910448 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:18.910407    4559 generic.go:356] "Generic (PLEG): container finished" podID="dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f" containerID="020710e28adc289961941ac338a8cac45e4168136eea0d1aae2421f3e826392d" exitCode=0
Oct 29 22:32:18.910917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:18.910503    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg" event={"ID":"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f","Type":"ContainerDied","Data":"020710e28adc289961941ac338a8cac45e4168136eea0d1aae2421f3e826392d"}
Oct 29 22:32:19.203303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:19.203267    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"
Oct 29 22:32:19.229506 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:19.229478    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-maas-api-openshift-ca\") pod \"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0\" (UID: \"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0\") "
Oct 29 22:32:19.229682 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:19.229612    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-kube-api-access-wqcdq\" (UniqueName: \"kubernetes.io/projected/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-kube-api-access-wqcdq\") pod \"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0\" (UID: \"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0\") "
Oct 29 22:32:19.229758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:19.229688    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-maas-api-xks-ca\") pod \"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0\" (UID: \"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0\") "
Oct 29 22:32:19.229885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:19.229857    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-maas-api-openshift-ca" pod "b44b9797-70d5-4a9e-a5bb-9b6afe189ab0" (UID: "b44b9797-70d5-4a9e-a5bb-9b6afe189ab0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:19.230262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:19.230237    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:19.232047 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:19.232024    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-maas-api-xks-ca" pod "b44b9797-70d5-4a9e-a5bb-9b6afe189ab0" (UID: "b44b9797-70d5-4a9e-a5bb-9b6afe189ab0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:19.232142 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:19.232094    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-kube-api-access-wqcdq" pod "b44b9797-70d5-4a9e-a5bb-9b6afe189ab0" (UID: "b44b9797-70d5-4a9e-a5bb-9b6afe189ab0"). InnerVolumeSpecName "kube-api-access-wqcdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:19.332640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:19.332609    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqcdq\" (UniqueName: \"kubernetes.io/projected/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-kube-api-access-wqcdq\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:19.332640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:19.332640    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:19.918899 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:19.918871    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"
Oct 29 22:32:19.919398 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:19.918862    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8" event={"ID":"b44b9797-70d5-4a9e-a5bb-9b6afe189ab0","Type":"ContainerDied","Data":"5b8bba117107bba5bc1e7fff0c94f58c875464453b401987c56965519a5f9003"}
Oct 29 22:32:19.919398 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:19.919034    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b8bba117107bba5bc1e7fff0c94f58c875464453b401987c56965519a5f9003"
Oct 29 22:32:19.921544 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:19.921508    4559 generic.go:356] "Generic (PLEG): container finished" podID="b15b2407-227c-40f8-95a9-4bfad483a2bb" containerID="2b0b55419a6269f1996fd04025fef6fff0f2f58852f95c980771fa74ea20f055" exitCode=0
Oct 29 22:32:19.921693 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:19.921585    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7" event={"ID":"b15b2407-227c-40f8-95a9-4bfad483a2bb","Type":"ContainerDied","Data":"2b0b55419a6269f1996fd04025fef6fff0f2f58852f95c980771fa74ea20f055"}
Oct 29 22:32:20.106426 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.106396    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"
Oct 29 22:32:20.143398 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.143370    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:32:20.143801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.143786    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-maas-api-xks-ca\") pod \"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f\" (UID: \"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f\") "
Oct 29 22:32:20.143920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.143903    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-kube-api-access-l26f2\" (UniqueName: \"kubernetes.io/projected/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-kube-api-access-l26f2\") pod \"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f\" (UID: \"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f\") "
Oct 29 22:32:20.147642 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.146091    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-maas-api-openshift-ca" pod "dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f" (UID: "dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:20.147642 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.147452    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-maas-api-openshift-ca\") pod \"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f\" (UID: \"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f\") "
Oct 29 22:32:20.148522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.148492    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-maas-api-xks-ca" pod "dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f" (UID: "dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:20.149256 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.149217    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:20.149256 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.149245    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:20.151463 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.151428    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-kube-api-access-l26f2" pod "dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f" (UID: "dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f"). InnerVolumeSpecName "kube-api-access-l26f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:20.250043 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.250003    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-token\" (UniqueName: \"kubernetes.io/projected/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-token\") pod \"b15b2407-227c-40f8-95a9-4bfad483a2bb\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") "
Oct 29 22:32:20.250241 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.250067    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-workload-socket\") pod \"b15b2407-227c-40f8-95a9-4bfad483a2bb\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") "
Oct 29 22:32:20.250241 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.250095    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b15b2407-227c-40f8-95a9-4bfad483a2bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15b2407-227c-40f8-95a9-4bfad483a2bb-istiod-ca-cert\") pod \"b15b2407-227c-40f8-95a9-4bfad483a2bb\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") "
Oct 29 22:32:20.250241 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.250197    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b15b2407-227c-40f8-95a9-4bfad483a2bb-kube-api-access-pszvl\" (UniqueName: \"kubernetes.io/projected/b15b2407-227c-40f8-95a9-4bfad483a2bb-kube-api-access-pszvl\") pod \"b15b2407-227c-40f8-95a9-4bfad483a2bb\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") "
Oct 29 22:32:20.250409 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.250264    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-data\") pod \"b15b2407-227c-40f8-95a9-4bfad483a2bb\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") "
Oct 29 22:32:20.250409 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.250302    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-envoy\") pod \"b15b2407-227c-40f8-95a9-4bfad483a2bb\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") "
Oct 29 22:32:20.250409 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.250345    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-credential-socket\") pod \"b15b2407-227c-40f8-95a9-4bfad483a2bb\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") "
Oct 29 22:32:20.250554 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.250446    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-podinfo\") pod \"b15b2407-227c-40f8-95a9-4bfad483a2bb\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") "
Oct 29 22:32:20.250554 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.250499    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-workload-certs\") pod \"b15b2407-227c-40f8-95a9-4bfad483a2bb\" (UID: \"b15b2407-227c-40f8-95a9-4bfad483a2bb\") "
Oct 29 22:32:20.250670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.250636    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-credential-socket" pod "b15b2407-227c-40f8-95a9-4bfad483a2bb" (UID: "b15b2407-227c-40f8-95a9-4bfad483a2bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:32:20.250720 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.250670    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b15b2407-227c-40f8-95a9-4bfad483a2bb-istiod-ca-cert" pod "b15b2407-227c-40f8-95a9-4bfad483a2bb" (UID: "b15b2407-227c-40f8-95a9-4bfad483a2bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:20.250771 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.250741    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-workload-socket" pod "b15b2407-227c-40f8-95a9-4bfad483a2bb" (UID: "b15b2407-227c-40f8-95a9-4bfad483a2bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:32:20.251025 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.251005    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-workload-certs" pod "b15b2407-227c-40f8-95a9-4bfad483a2bb" (UID: "b15b2407-227c-40f8-95a9-4bfad483a2bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:32:20.251297 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.251280    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:20.251366 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.251301    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15b2407-227c-40f8-95a9-4bfad483a2bb-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:20.251366 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.251314    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:20.251366 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.251327    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:20.251366 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.251341    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l26f2\" (UniqueName: \"kubernetes.io/projected/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f-kube-api-access-l26f2\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:20.251700 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.251675    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-data" pod "b15b2407-227c-40f8-95a9-4bfad483a2bb" (UID: "b15b2407-227c-40f8-95a9-4bfad483a2bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:32:20.252919 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.252888    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15b2407-227c-40f8-95a9-4bfad483a2bb-kube-api-access-pszvl" pod "b15b2407-227c-40f8-95a9-4bfad483a2bb" (UID: "b15b2407-227c-40f8-95a9-4bfad483a2bb"). InnerVolumeSpecName "kube-api-access-pszvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:20.253554 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.253534    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-podinfo" pod "b15b2407-227c-40f8-95a9-4bfad483a2bb" (UID: "b15b2407-227c-40f8-95a9-4bfad483a2bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:32:20.254019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.253993    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-envoy" pod "b15b2407-227c-40f8-95a9-4bfad483a2bb" (UID: "b15b2407-227c-40f8-95a9-4bfad483a2bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:32:20.254019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.254003    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-token" pod "b15b2407-227c-40f8-95a9-4bfad483a2bb" (UID: "b15b2407-227c-40f8-95a9-4bfad483a2bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:20.353490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.353450    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:20.353490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.353490    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:20.353707 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.353506    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:20.353707 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.353522    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pszvl\" (UniqueName: \"kubernetes.io/projected/b15b2407-227c-40f8-95a9-4bfad483a2bb-kube-api-access-pszvl\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:20.353707 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.353535    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15b2407-227c-40f8-95a9-4bfad483a2bb-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:32:20.931091 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.931058    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"
Oct 29 22:32:20.931724 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.931094    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg" event={"ID":"dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f","Type":"ContainerDied","Data":"b913cf23b3b36e6c96d452549b4fc82b04cae7ac74f520c9839eed0ad0379a3b"}
Oct 29 22:32:20.931724 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.931148    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b913cf23b3b36e6c96d452549b4fc82b04cae7ac74f520c9839eed0ad0379a3b"
Oct 29 22:32:20.932916 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.932893    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"
Oct 29 22:32:20.933065 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.932912    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7" event={"ID":"b15b2407-227c-40f8-95a9-4bfad483a2bb","Type":"ContainerDied","Data":"7daed353eda6682315cf42a70d04f76cc4014182a06a0a9c9eed297f9cc3ae8c"}
Oct 29 22:32:20.933065 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:20.932987    4559 scope.go:122] "RemoveContainer" containerID="2b0b55419a6269f1996fd04025fef6fff0f2f58852f95c980771fa74ea20f055"
Oct 29 22:32:21.014261 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.014228    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"]
Oct 29 22:32:21.018139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.018112    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-k9fn7"]
Oct 29 22:32:21.238592 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.238511    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b15b2407-227c-40f8-95a9-4bfad483a2bb" path="/var/lib/kubelet/pods/b15b2407-227c-40f8-95a9-4bfad483a2bb/volumes"
Oct 29 22:32:21.289433 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.289393    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"]
Oct 29 22:32:21.290188 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.290168    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b44b9797-70d5-4a9e-a5bb-9b6afe189ab0" containerName="revoke-keys"
Oct 29 22:32:21.290188 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.290189    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44b9797-70d5-4a9e-a5bb-9b6afe189ab0" containerName="revoke-keys"
Oct 29 22:32:21.290307 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.290201    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b15b2407-227c-40f8-95a9-4bfad483a2bb" containerName="istio-proxy"
Oct 29 22:32:21.290307 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.290207    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15b2407-227c-40f8-95a9-4bfad483a2bb" containerName="istio-proxy"
Oct 29 22:32:21.290307 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.290225    4559 cpu_manager.go:414] "containerMap: removing container" podUID="dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f" containerName="revoke-keys"
Oct 29 22:32:21.290307 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.290230    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f" containerName="revoke-keys"
Oct 29 22:32:21.290427 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.290325    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44b9797-70d5-4a9e-a5bb-9b6afe189ab0" containerName="revoke-keys"
Oct 29 22:32:21.290427 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.290334    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15b2407-227c-40f8-95a9-4bfad483a2bb" containerName="istio-proxy"
Oct 29 22:32:21.290427 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.290346    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f" containerName="revoke-keys"
Oct 29 22:32:21.295435 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.295410    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.318681 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.318648    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"]
Oct 29 22:32:21.373206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.373167    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.373396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.373332    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee61a67-ee9f-4311-9042-9920dea1d942-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.373396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.373368    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.373530 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.373457    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.373530 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.373506    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.373640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.373555    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.373640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.373572    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.373725 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.373694    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.373774 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.373755    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfvq5\" (UniqueName: \"kubernetes.io/projected/7ee61a67-ee9f-4311-9042-9920dea1d942-kube-api-access-jfvq5\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.474861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.474823    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee61a67-ee9f-4311-9042-9920dea1d942-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.475078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.474872    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.475078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.474943    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.475078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.474985    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.475227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.475151    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.475227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.475197    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.475326 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.475248    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.475326 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.475292    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvq5\" (UniqueName: \"kubernetes.io/projected/7ee61a67-ee9f-4311-9042-9920dea1d942-kube-api-access-jfvq5\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.475438 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.475366    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.475438 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.475395    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.475614 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.475438    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.475671 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.475611    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.475725 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.475692    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee61a67-ee9f-4311-9042-9920dea1d942-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.475725 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.475716    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.477639 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.477608    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.477988 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.477969    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.495238 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.495162    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.495347 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.495299    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfvq5\" (UniqueName: \"kubernetes.io/projected/7ee61a67-ee9f-4311-9042-9920dea1d942-kube-api-access-jfvq5\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.611303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.611267    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:21.937958 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.937926    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"]
Oct 29 22:32:21.938311 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:32:21.938077    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee61a67_ee9f_4311_9042_9920dea1d942.slice/crio-8b6b7e612c834cd15e03d3c81d9fa319f438bbe9687d30f64625aa154beba04d WatchSource:0}: Error finding container 8b6b7e612c834cd15e03d3c81d9fa319f438bbe9687d30f64625aa154beba04d: Status 404 returned error can't find the container with id 8b6b7e612c834cd15e03d3c81d9fa319f438bbe9687d30f64625aa154beba04d
Oct 29 22:32:21.940972 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.940933    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:32:21.941052 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.941024    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:32:21.941089 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:21.941067    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:32:22.963499 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:22.963463    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn" event={"ID":"7ee61a67-ee9f-4311-9042-9920dea1d942","Type":"ContainerStarted","Data":"14f60b1c66e8127054548dcea75a18e55ccac104fdf5e25e7eedbdd142ff0679"}
Oct 29 22:32:22.963932 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:22.963529    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn" event={"ID":"7ee61a67-ee9f-4311-9042-9920dea1d942","Type":"ContainerStarted","Data":"8b6b7e612c834cd15e03d3c81d9fa319f438bbe9687d30f64625aa154beba04d"}
Oct 29 22:32:23.062834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:23.062263    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn" podStartSLOduration=2.062240952 podStartE2EDuration="2.062240952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:23.019975867 +0000 UTC m=+1081.931419428" watchObservedRunningTime="2026-10-29 22:32:23.062240952 +0000 UTC m=+1081.973684509"
Oct 29 22:32:23.611632 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:23.611592    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:23.613549 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:23.613519    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:32:23.613713 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:23.613581    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn" podUID="7ee61a67-ee9f-4311-9042-9920dea1d942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 22:32:24.612224 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:24.612185    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:32:24.612644 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:24.612254    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn" podUID="7ee61a67-ee9f-4311-9042-9920dea1d942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 22:32:25.612573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:25.612532    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:32:25.613041 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:25.612602    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn" podUID="7ee61a67-ee9f-4311-9042-9920dea1d942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 22:32:26.612158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:26.612111    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:32:26.612347 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:26.612174    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn" podUID="7ee61a67-ee9f-4311-9042-9920dea1d942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 22:32:27.612593 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:27.612555    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:32:27.613144 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:27.612610    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn" podUID="7ee61a67-ee9f-4311-9042-9920dea1d942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 22:32:28.611830 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:28.611769    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:32:28.611993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:28.611858    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn" podUID="7ee61a67-ee9f-4311-9042-9920dea1d942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 22:32:29.686428 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:29.686391    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:29.687439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:29.687396    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:29.688638 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:29.688611    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:32:32.937149 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:32.937112    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:32:35.488968 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:35.488924    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:32:35.489595 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:35.489561    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:35.971838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:35.959287    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:32:35.972060 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:35.971907    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.141849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:36.141179    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:32:36.141849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:36.141247    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" podUID="b616ef83-09fe-47a2-9d15-4ba5a0126998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.578838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:36.565924    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.578838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:36.568224    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.622838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:36.621840    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.669839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:36.664447    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.818419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:36.816952    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 22:32:36.978838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:36.973993    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:32:36.978838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:36.975135    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:32:37.003843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:37.001999    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:37.003843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:37.002700    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:32:37.330112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:37.308909    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:37.588556 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:37.588444    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:37.671443 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:37.628355    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:32:37.688934 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:37.637471    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:37.697000 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:37.671870    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:32:37.697000 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:37.694309    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:32:37.697000 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:37.694350    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:32:37.825990 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:37.825938    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:37.839583 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:37.839463    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:32:37.977947 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:37.977902    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:38.072023 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:38.071978    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:38.686839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:38.676563    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:38.692527 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:38.692338    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:38.727859 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:38.692370    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:32:38.733956 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:38.733916    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:32:38.735044 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:38.735008    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:38.831505 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:38.831452    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 22:32:39.035896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:39.034470    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:32:39.076838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:39.072395    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.130932 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:39.130893    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Oct 29 22:32:39.750436 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:39.750219    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.753031 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:39.752979    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.796857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:39.780829    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.859850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:39.847032    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:40.084901 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:40.047207    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:40.152840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:40.142924    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:32:40.443963 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:40.443915    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:40.774030 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:40.773926    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Oct 29 22:32:40.776129 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:40.776080    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:40.784436 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:40.777222    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:32:40.785331 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:40.785292    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:32:40.847383 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:40.847329    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:41.090080 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:41.088928    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:41.151854 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:41.145931    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:32:41.724846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:41.724053    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:32:41.802837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:41.801914    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:41.806952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:41.805707    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:32:41.821146 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:41.808678    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:32:41.903971 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:41.903910    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:42.215888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:42.148929    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:42.498903 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:42.498620    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:32:42.597796 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:42.595534    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:32:42.668366 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:42.668335    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:32:42.747135 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:42.736912    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:32:42.747135 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:42.745034    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:32:43.784461 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:32:43.784420    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:32:57.705883 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.705845    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"]
Oct 29 22:32:57.715422 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.715398    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.740462 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.740415    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-dc369ed0-openshift-default-dockercfg-62qww\""
Oct 29 22:32:57.775354 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.775282    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"]
Oct 29 22:32:57.832837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.812837    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtrrd\" (UniqueName: \"kubernetes.io/projected/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-kube-api-access-qtrrd\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.833196 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.833165    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-workload-certs\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.833444 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.833427    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istiod-ca-cert\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.833659 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.833642    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-token\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.853171 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.833801    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-workload-socket\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.853535 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.853500    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-envoy\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.857838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.853764    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-credential-socket\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.868031 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.867981    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-podinfo\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.888514 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.869053    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-data\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.980905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.971887    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-credential-socket\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.980905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.971988    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-podinfo\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.980905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.972046    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-data\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.980905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.972148    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtrrd\" (UniqueName: \"kubernetes.io/projected/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-kube-api-access-qtrrd\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.980905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.972177    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-workload-certs\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.980905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.972258    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istiod-ca-cert\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.980905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.972334    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-token\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.980905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.972359    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-workload-socket\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:57.980905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.972449    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-envoy\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:58.042538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:57.983105    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-credential-socket\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:58.042538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:58.004182    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istiod-ca-cert\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:58.042538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:58.004435    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-workload-certs\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:58.042538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:58.013268    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-envoy\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:58.042538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:58.013568    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-workload-socket\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:58.042538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:58.013885    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-data\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:58.080028 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:58.079988    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-podinfo\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:58.098107 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:58.080601    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtrrd\" (UniqueName: \"kubernetes.io/projected/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-kube-api-access-qtrrd\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:58.098107 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:58.092074    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-token\") pod \"e2e-ait-dc369ed0-openshift-default-858c684c57-npckw\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") " pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:58.342349 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:58.341089    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:32:58.764616 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:58.763067    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"]
Oct 29 22:32:58.774687 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:58.772460    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:32:58.774687 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:58.772577    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:32:58.774687 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:58.772636    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:32:59.789927 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:59.789866    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw" event={"ID":"10c127bb-5410-4a3c-974b-0b7f4fb31ccb","Type":"ContainerStarted","Data":"640249050da05295c9b701c937b530af139c560d31b271fadc8d42fd7af979b1"}
Oct 29 22:32:59.798899 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:59.794983    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw" event={"ID":"10c127bb-5410-4a3c-974b-0b7f4fb31ccb","Type":"ContainerStarted","Data":"d9afa90759a55b933420bd772b178ce40d6df4ce07f57fcb997a640d1101e4cb"}
Oct 29 22:32:59.870834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:32:59.867756    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw" podStartSLOduration=2.867732117 podStartE2EDuration="2.867732117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:59.860341188 +0000 UTC m=+1118.771784773" watchObservedRunningTime="2026-10-29 22:32:59.867732117 +0000 UTC m=+1118.779175671"
Oct 29 22:33:00.364833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:00.349480    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:33:00.391837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:00.391357    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:33:00.845278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:00.845249    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:33:00.875162 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:00.875127    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:33:01.987091 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:01.987047    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:02.065193 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:02.060842    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:33:02.169125 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:02.167490    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 22:33:02.273663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:02.247373    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:33:02.273663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:02.247453    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl" podUID="220a1952-3291-485e-b6d4-4336b1cd1db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:02.511989 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:02.511751    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:02.911839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:02.859853    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:33:02.914618 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:02.864752    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:33:02.916004 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:33:02.876995    4559 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.607s"
Oct 29 22:33:02.933892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:02.883125    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:33:02.933892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:02.931069    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82" podUID="6a3bfdf3-5fca-44b9-88d7-bf525139ee93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:03.022837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:03.000676    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:03.167797 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:03.093467    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:33:03.209291 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:03.173154    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:33:03.209612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:03.209593    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:33:03.250139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:03.179897    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 22:33:03.292865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:03.222114    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:03.587188 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:03.587093    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:33:03.723855 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:03.720935    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:33:03.723855 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:03.720993    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:33:03.775184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:03.775154    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:33:03.983194 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:03.971781    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:04.031844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:04.006233    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:33:04.068317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:04.068287    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:33:04.121888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:04.121860    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:33:04.133132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:04.133106    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:33:04.157509 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:04.157483    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:33:04.963922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:04.963878    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"]
Oct 29 22:33:04.987118 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:04.987085    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"]
Oct 29 22:33:04.987528 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:04.987440    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"
Oct 29 22:33:05.017286 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:05.015777    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-dc369ed0-dockercfg-mszw4\""
Oct 29 22:33:05.034165 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:05.034134    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-dc369ed0\""
Oct 29 22:33:05.121924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:05.121874    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2ch4\" (UniqueName: \"kubernetes.io/projected/dc75ea97-6e3a-4c05-aad3-9e83e5cf240d-kube-api-access-m2ch4\") pod \"payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf\" (UID: \"dc75ea97-6e3a-4c05-aad3-9e83e5cf240d\") " pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"
Oct 29 22:33:05.122374 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:05.122350    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc75ea97-6e3a-4c05-aad3-9e83e5cf240d-plugins-config-volume\") pod \"payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf\" (UID: \"dc75ea97-6e3a-4c05-aad3-9e83e5cf240d\") " pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"
Oct 29 22:33:05.223719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:05.223626    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2ch4\" (UniqueName: \"kubernetes.io/projected/dc75ea97-6e3a-4c05-aad3-9e83e5cf240d-kube-api-access-m2ch4\") pod \"payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf\" (UID: \"dc75ea97-6e3a-4c05-aad3-9e83e5cf240d\") " pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"
Oct 29 22:33:05.223903 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:05.223830    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc75ea97-6e3a-4c05-aad3-9e83e5cf240d-plugins-config-volume\") pod \"payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf\" (UID: \"dc75ea97-6e3a-4c05-aad3-9e83e5cf240d\") " pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"
Oct 29 22:33:05.224589 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:05.224569    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc75ea97-6e3a-4c05-aad3-9e83e5cf240d-plugins-config-volume\") pod \"payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf\" (UID: \"dc75ea97-6e3a-4c05-aad3-9e83e5cf240d\") " pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"
Oct 29 22:33:05.251421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:05.251382    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2ch4\" (UniqueName: \"kubernetes.io/projected/dc75ea97-6e3a-4c05-aad3-9e83e5cf240d-kube-api-access-m2ch4\") pod \"payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf\" (UID: \"dc75ea97-6e3a-4c05-aad3-9e83e5cf240d\") " pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"
Oct 29 22:33:05.343095 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:05.343045    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"
Oct 29 22:33:05.669974 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:05.669935    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"]
Oct 29 22:33:06.142272 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.141247    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"]
Oct 29 22:33:06.148099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.148073    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.170338 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.170306    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"]
Oct 29 22:33:06.273204 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.273173    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.273424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.273253    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.273424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.273366    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.273424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.273399    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.273424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.273419    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.273661 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.273450    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmgtm\" (UniqueName: \"kubernetes.io/projected/4efa9443-3039-48cf-ad64-6d36ee0e1335-kube-api-access-nmgtm\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.273661 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.273474    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.273661 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.273514    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.273661 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.273551    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efa9443-3039-48cf-ad64-6d36ee0e1335-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.280437 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.280400    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf" event={"ID":"dc75ea97-6e3a-4c05-aad3-9e83e5cf240d","Type":"ContainerStarted","Data":"d4029bf3bb2c343617a8e84cffea9124e159d5da360229cbfa22a2da2ec4d9cd"}
Oct 29 22:33:06.280582 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.280469    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf" event={"ID":"dc75ea97-6e3a-4c05-aad3-9e83e5cf240d","Type":"ContainerStarted","Data":"1f0647d047b82a753c2ec2221fec5666b54f5e2b23d47d97ccf2dcef3d7544a6"}
Oct 29 22:33:06.280582 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.280495    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"
Oct 29 22:33:06.319712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.319656    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf" podStartSLOduration=2.319642753 podStartE2EDuration="2.319642753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:06.318359889 +0000 UTC m=+1125.229803443" watchObservedRunningTime="2026-10-29 22:33:06.319642753 +0000 UTC m=+1125.231086308"
Oct 29 22:33:06.375418 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.375359    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.375614 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.375487    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.375614 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.375593    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.375752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.375644    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.375752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.375670    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.375752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.375744    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmgtm\" (UniqueName: \"kubernetes.io/projected/4efa9443-3039-48cf-ad64-6d36ee0e1335-kube-api-access-nmgtm\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.375923 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.375789    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.375923 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.375860    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.376017 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.375940    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efa9443-3039-48cf-ad64-6d36ee0e1335-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.376387 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.376341    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.376674 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.376645    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.376846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.376670    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.376846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.376690    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.377207 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.377182    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efa9443-3039-48cf-ad64-6d36ee0e1335-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.378450 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.378426    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.378929 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.378906    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.396148 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.396076    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.396517 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.396494    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmgtm\" (UniqueName: \"kubernetes.io/projected/4efa9443-3039-48cf-ad64-6d36ee0e1335-kube-api-access-nmgtm\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.477905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.472098    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:06.862246 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.862214    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"]
Oct 29 22:33:06.862642 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:33:06.862596    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efa9443_3039_48cf_ad64_6d36ee0e1335.slice/crio-2bd091ec9211b534addad83c8a906c0bb7a9a5b32141682003bfa0b830df3253 WatchSource:0}: Error finding container 2bd091ec9211b534addad83c8a906c0bb7a9a5b32141682003bfa0b830df3253: Status 404 returned error can't find the container with id 2bd091ec9211b534addad83c8a906c0bb7a9a5b32141682003bfa0b830df3253
Oct 29 22:33:06.865247 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.865212    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:33:06.865345 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.865280    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:33:06.865345 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:06.865309    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:33:07.290019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:07.289981    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q" event={"ID":"4efa9443-3039-48cf-ad64-6d36ee0e1335","Type":"ContainerStarted","Data":"e7ed50ad58560238b27737a4f2199e48f808d2aac656ba114b751327c735e8a4"}
Oct 29 22:33:07.290425 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:07.290044    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q" event={"ID":"4efa9443-3039-48cf-ad64-6d36ee0e1335","Type":"ContainerStarted","Data":"2bd091ec9211b534addad83c8a906c0bb7a9a5b32141682003bfa0b830df3253"}
Oct 29 22:33:07.339034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:07.338974    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q" podStartSLOduration=1.338960427 podStartE2EDuration="1.338960427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:07.334326183 +0000 UTC m=+1126.245769736" watchObservedRunningTime="2026-10-29 22:33:07.338960427 +0000 UTC m=+1126.250404046"
Oct 29 22:33:07.473647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:07.473606    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:07.475322 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:07.475293    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:33:07.475464 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:07.475344    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q" podUID="4efa9443-3039-48cf-ad64-6d36ee0e1335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 29 22:33:08.473290 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:08.473251    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:33:08.473956 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:08.473920    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q" podUID="4efa9443-3039-48cf-ad64-6d36ee0e1335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 29 22:33:09.473461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:09.473425    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:33:09.473912 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:09.473487    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q" podUID="4efa9443-3039-48cf-ad64-6d36ee0e1335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 29 22:33:10.473497 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:10.473463    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:33:10.474125 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:10.474095    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q" podUID="4efa9443-3039-48cf-ad64-6d36ee0e1335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 29 22:33:11.473362 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:11.473329    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:33:11.473550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:11.473387    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q" podUID="4efa9443-3039-48cf-ad64-6d36ee0e1335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 29 22:33:12.472979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:12.472943    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:33:12.473172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:12.473008    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q" podUID="4efa9443-3039-48cf-ad64-6d36ee0e1335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 29 22:33:13.475295 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:13.475239    4559 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:33:13.475935 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:13.475898    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q" podUID="4efa9443-3039-48cf-ad64-6d36ee0e1335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 29 22:33:13.695416 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:33:13.695374    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:33:14.477912 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:14.477880    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:14.478319 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:14.478299    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:14.478985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:14.478966    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:33:15.008488 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.008456    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"]
Oct 29 22:33:15.012876 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.012855    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:15.034184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.034146    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f7dea32d-serving-cert\""
Oct 29 22:33:15.038026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.037993    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"]
Oct 29 22:33:15.049975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.049922    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66471fb0-5c91-4a44-aac0-23361a42c852-openshift-service-ca\") pod \"maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf\" (UID: \"66471fb0-5c91-4a44-aac0-23361a42c852\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:15.050162 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.049986    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jrzz\" (UniqueName: \"kubernetes.io/projected/66471fb0-5c91-4a44-aac0-23361a42c852-kube-api-access-2jrzz\") pod \"maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf\" (UID: \"66471fb0-5c91-4a44-aac0-23361a42c852\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:15.050233 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.050050    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66471fb0-5c91-4a44-aac0-23361a42c852-metrics-certs\") pod \"maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf\" (UID: \"66471fb0-5c91-4a44-aac0-23361a42c852\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:15.050830 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.050409    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66471fb0-5c91-4a44-aac0-23361a42c852-maas-api-tls\") pod \"maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf\" (UID: \"66471fb0-5c91-4a44-aac0-23361a42c852\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:15.151529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.151473    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jrzz\" (UniqueName: \"kubernetes.io/projected/66471fb0-5c91-4a44-aac0-23361a42c852-kube-api-access-2jrzz\") pod \"maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf\" (UID: \"66471fb0-5c91-4a44-aac0-23361a42c852\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:15.151739 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.151569    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66471fb0-5c91-4a44-aac0-23361a42c852-metrics-certs\") pod \"maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf\" (UID: \"66471fb0-5c91-4a44-aac0-23361a42c852\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:15.151825 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.151775    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66471fb0-5c91-4a44-aac0-23361a42c852-maas-api-tls\") pod \"maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf\" (UID: \"66471fb0-5c91-4a44-aac0-23361a42c852\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:15.151926 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:33:15.151906    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-serving-cert: secret "maas-api-e2e-ait-f7dea32d-serving-cert" not found
Oct 29 22:33:15.151992 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.151938    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66471fb0-5c91-4a44-aac0-23361a42c852-openshift-service-ca\") pod \"maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf\" (UID: \"66471fb0-5c91-4a44-aac0-23361a42c852\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:15.151992 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:33:15.151982    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66471fb0-5c91-4a44-aac0-23361a42c852-maas-api-tls podName:66471fb0-5c91-4a44-aac0-23361a42c852 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:15.651958409 +0000 UTC m=+1134.563401948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66471fb0-5c91-4a44-aac0-23361a42c852-maas-api-tls") pod "maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf" (UID: "66471fb0-5c91-4a44-aac0-23361a42c852") : secret "maas-api-e2e-ait-f7dea32d-serving-cert" not found
Oct 29 22:33:15.152671 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.152642    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66471fb0-5c91-4a44-aac0-23361a42c852-openshift-service-ca\") pod \"maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf\" (UID: \"66471fb0-5c91-4a44-aac0-23361a42c852\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:15.154417 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.154394    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66471fb0-5c91-4a44-aac0-23361a42c852-metrics-certs\") pod \"maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf\" (UID: \"66471fb0-5c91-4a44-aac0-23361a42c852\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:15.172366 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.172341    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jrzz\" (UniqueName: \"kubernetes.io/projected/66471fb0-5c91-4a44-aac0-23361a42c852-kube-api-access-2jrzz\") pod \"maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf\" (UID: \"66471fb0-5c91-4a44-aac0-23361a42c852\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:15.671091 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.671007    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66471fb0-5c91-4a44-aac0-23361a42c852-maas-api-tls\") pod \"maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf\" (UID: \"66471fb0-5c91-4a44-aac0-23361a42c852\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:15.673662 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.673633    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66471fb0-5c91-4a44-aac0-23361a42c852-maas-api-tls\") pod \"maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf\" (UID: \"66471fb0-5c91-4a44-aac0-23361a42c852\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:15.936889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:15.932241    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:16.349241 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:16.344220    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:33:16.353845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:16.351273    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"]
Oct 29 22:33:16.442964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:16.442924    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf" event={"ID":"66471fb0-5c91-4a44-aac0-23361a42c852","Type":"ContainerStarted","Data":"faa04fde41b39bc028db8be5eeb4bd5a384909f36900eed1c9ae99c95c79d3c5"}
Oct 29 22:33:17.293838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:17.293778    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"
Oct 29 22:33:17.463739 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:17.463702    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf" event={"ID":"66471fb0-5c91-4a44-aac0-23361a42c852","Type":"ContainerStarted","Data":"c3de853ae45134e2aff1dc84e20e41a2f0fe85c7ff90736bb26d6ca20fcd1c61"}
Oct 29 22:33:17.463943 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:17.463793    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:17.505311 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:17.505258    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf" podStartSLOduration=2.996614025 podStartE2EDuration="3.505241317s" totalImagesPullingTime="508.627292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:17.504250221 +0000 UTC m=+1136.415693789" watchObservedRunningTime="2026-10-29 22:33:17.505241317 +0000 UTC m=+1136.416684871"
Oct 29 22:33:18.391714 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:18.391669    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"]
Oct 29 22:33:18.396488 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:18.396463    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:33:18.419182 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:18.419149    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"]
Oct 29 22:33:18.429703 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:18.424369    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-dc369ed0-serving-cert\""
Oct 29 22:33:18.475604 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:18.475569    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x2zs\" (UniqueName: \"kubernetes.io/projected/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-kube-api-access-8x2zs\") pod \"maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg\" (UID: \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:33:18.475777 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:18.475668    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-openshift-service-ca\") pod \"maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg\" (UID: \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:33:18.475777 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:18.475710    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-maas-api-tls\") pod \"maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg\" (UID: \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:33:18.475777 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:18.475757    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-metrics-certs\") pod \"maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg\" (UID: \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:33:18.589052 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:18.588097    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x2zs\" (UniqueName: \"kubernetes.io/projected/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-kube-api-access-8x2zs\") pod \"maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg\" (UID: \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:33:18.589052 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:18.588222    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-openshift-service-ca\") pod \"maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg\" (UID: \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:33:18.589052 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:18.588267    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-maas-api-tls\") pod \"maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg\" (UID: \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:33:18.589052 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:18.588323    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-metrics-certs\") pod \"maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg\" (UID: \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:33:18.622577 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:18.622545    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-openshift-service-ca\") pod \"maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg\" (UID: \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:33:18.636965 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:18.636892    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-maas-api-tls\") pod \"maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg\" (UID: \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:33:18.665515 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:18.665434    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-metrics-certs\") pod \"maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg\" (UID: \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:33:18.684164 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:18.684071    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x2zs\" (UniqueName: \"kubernetes.io/projected/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-kube-api-access-8x2zs\") pod \"maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg\" (UID: \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:33:18.716445 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:18.716104    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:33:19.034322 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:19.034297    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"]
Oct 29 22:33:19.035630 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:33:19.035584    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076c0bce_da05_40aa_a50f_ce3ca7b5e6c9.slice/crio-b781fc81d9882af36e3a9d78b746a8de7fabcec66ab2bc1b4527740e7911608e WatchSource:0}: Error finding container b781fc81d9882af36e3a9d78b746a8de7fabcec66ab2bc1b4527740e7911608e: Status 404 returned error can't find the container with id b781fc81d9882af36e3a9d78b746a8de7fabcec66ab2bc1b4527740e7911608e
Oct 29 22:33:19.486113 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:19.486074    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg" event={"ID":"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9","Type":"ContainerStarted","Data":"008db6eea3bb936fb2b98e8acd6701e2f4614ba5f3b0ccdecd3852e8e7428041"}
Oct 29 22:33:19.486464 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:19.486142    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg" event={"ID":"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9","Type":"ContainerStarted","Data":"b781fc81d9882af36e3a9d78b746a8de7fabcec66ab2bc1b4527740e7911608e"}
Oct 29 22:33:19.486464 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:19.486175    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:33:19.529361 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:19.529308    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg" podStartSLOduration=1.156234645 podStartE2EDuration="1.529294221s" totalImagesPullingTime="373.059576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:19.526205153 +0000 UTC m=+1138.437648746" watchObservedRunningTime="2026-10-29 22:33:19.529294221 +0000 UTC m=+1138.440737775"
Oct 29 22:33:23.552985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:23.551715    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:25.525667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:25.525626    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:33:25.556018 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:25.555987    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:33:35.622520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.622476    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"]
Oct 29 22:33:35.628349 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.628321    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.663731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.663692    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"]
Oct 29 22:33:35.815268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.815235    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z7fq\" (UniqueName: \"kubernetes.io/projected/cd055ebb-b30b-447d-831d-1ff95c57dc55-kube-api-access-4z7fq\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.815467 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.815279    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.815467 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.815301    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.815467 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.815377    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.815467 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.815431    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.815467 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.815449    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd055ebb-b30b-447d-831d-1ff95c57dc55-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.815644 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.815537    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.815644 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.815558    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.815644 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.815581    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.916904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.916860    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.919317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.919283    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z7fq\" (UniqueName: \"kubernetes.io/projected/cd055ebb-b30b-447d-831d-1ff95c57dc55-kube-api-access-4z7fq\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.924958 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.917570    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.925151 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.924912    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.925151 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.925043    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.925151 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.925098    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.925410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.925159    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.925410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.925183    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd055ebb-b30b-447d-831d-1ff95c57dc55-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.925410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.925366    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.925410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.925407    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.926031 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.926001    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.926281 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.926255    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.940836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.926958    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd055ebb-b30b-447d-831d-1ff95c57dc55-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.940836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.927190    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.959787 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.959746    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.979393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.975044    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.979393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.976697    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:35.983142 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:35.983080    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z7fq\" (UniqueName: \"kubernetes.io/projected/cd055ebb-b30b-447d-831d-1ff95c57dc55-kube-api-access-4z7fq\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:36.254267 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:36.254174    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:36.586138 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:33:36.586001    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd055ebb_b30b_447d_831d_1ff95c57dc55.slice/crio-d3f02b1407e25e12cea401d385c42a2fe7bedf5cd87b73a6a0be05023407112e WatchSource:0}: Error finding container d3f02b1407e25e12cea401d385c42a2fe7bedf5cd87b73a6a0be05023407112e: Status 404 returned error can't find the container with id d3f02b1407e25e12cea401d385c42a2fe7bedf5cd87b73a6a0be05023407112e
Oct 29 22:33:36.586261 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:36.586217    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"]
Oct 29 22:33:36.589094 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:36.589061    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:33:36.589200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:36.589130    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:33:36.589200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:36.589158    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:33:36.801383 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:36.801332    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2" event={"ID":"cd055ebb-b30b-447d-831d-1ff95c57dc55","Type":"ContainerStarted","Data":"0f43e371f53cc199d3fd92e97bccd6c9e2c18f01f100998e4b9e6f9c59a92cd9"}
Oct 29 22:33:36.801868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:36.801447    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2" event={"ID":"cd055ebb-b30b-447d-831d-1ff95c57dc55","Type":"ContainerStarted","Data":"d3f02b1407e25e12cea401d385c42a2fe7bedf5cd87b73a6a0be05023407112e"}
Oct 29 22:33:36.850742 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:36.850618    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2" podStartSLOduration=1.8505988960000002 podStartE2EDuration="1.850598896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:36.850142243 +0000 UTC m=+1155.761585821" watchObservedRunningTime="2026-10-29 22:33:36.850598896 +0000 UTC m=+1155.762042450"
Oct 29 22:33:37.258845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:37.258679    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:37.259064 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:37.258845    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:33:37.259064 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:37.258895    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2" podUID="cd055ebb-b30b-447d-831d-1ff95c57dc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 22:33:38.258282 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:38.258239    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:33:38.258790 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:38.258310    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2" podUID="cd055ebb-b30b-447d-831d-1ff95c57dc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 22:33:39.258837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:39.257921    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:33:39.258837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:39.257994    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2" podUID="cd055ebb-b30b-447d-831d-1ff95c57dc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 22:33:40.258835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:40.258280    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:33:40.258835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:40.258358    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2" podUID="cd055ebb-b30b-447d-831d-1ff95c57dc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 22:33:41.282654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:41.280908    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:33:41.282654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:41.280970    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2" podUID="cd055ebb-b30b-447d-831d-1ff95c57dc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 22:33:42.256454 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:42.256405    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:33:42.256772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:42.256733    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2" podUID="cd055ebb-b30b-447d-831d-1ff95c57dc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 22:33:43.255119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:43.255087    4559 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:33:43.255616 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:43.255586    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2" podUID="cd055ebb-b30b-447d-831d-1ff95c57dc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 22:33:43.697087 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:33:43.697042    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:33:44.396602 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:44.396567    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:44.397236 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:44.397214    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:44.398246 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:44.398222    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:33:48.976824 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:48.976768    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"]
Oct 29 22:33:48.977716 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:48.977655    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf" podUID="66471fb0-5c91-4a44-aac0-23361a42c852" containerName="maas-api" containerID="cri-o://c3de853ae45134e2aff1dc84e20e41a2f0fe85c7ff90736bb26d6ca20fcd1c61" gracePeriod=30
Oct 29 22:33:49.411565 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:49.411533    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:49.525706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:49.475661    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66471fb0-5c91-4a44-aac0-23361a42c852-kube-api-access-2jrzz\" (UniqueName: \"kubernetes.io/projected/66471fb0-5c91-4a44-aac0-23361a42c852-kube-api-access-2jrzz\") pod \"66471fb0-5c91-4a44-aac0-23361a42c852\" (UID: \"66471fb0-5c91-4a44-aac0-23361a42c852\") "
Oct 29 22:33:49.525706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:49.475780    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66471fb0-5c91-4a44-aac0-23361a42c852-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66471fb0-5c91-4a44-aac0-23361a42c852-openshift-service-ca\") pod \"66471fb0-5c91-4a44-aac0-23361a42c852\" (UID: \"66471fb0-5c91-4a44-aac0-23361a42c852\") "
Oct 29 22:33:49.525706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:49.475889    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66471fb0-5c91-4a44-aac0-23361a42c852-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66471fb0-5c91-4a44-aac0-23361a42c852-metrics-certs\") pod \"66471fb0-5c91-4a44-aac0-23361a42c852\" (UID: \"66471fb0-5c91-4a44-aac0-23361a42c852\") "
Oct 29 22:33:49.525706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:49.476058    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66471fb0-5c91-4a44-aac0-23361a42c852-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66471fb0-5c91-4a44-aac0-23361a42c852-maas-api-tls\") pod \"66471fb0-5c91-4a44-aac0-23361a42c852\" (UID: \"66471fb0-5c91-4a44-aac0-23361a42c852\") "
Oct 29 22:33:49.525706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:49.489228    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66471fb0-5c91-4a44-aac0-23361a42c852-openshift-service-ca" pod "66471fb0-5c91-4a44-aac0-23361a42c852" (UID: "66471fb0-5c91-4a44-aac0-23361a42c852"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:33:49.525706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:49.493954    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66471fb0-5c91-4a44-aac0-23361a42c852-kube-api-access-2jrzz" pod "66471fb0-5c91-4a44-aac0-23361a42c852" (UID: "66471fb0-5c91-4a44-aac0-23361a42c852"). InnerVolumeSpecName "kube-api-access-2jrzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:33:49.525706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:49.496293    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66471fb0-5c91-4a44-aac0-23361a42c852-metrics-certs" pod "66471fb0-5c91-4a44-aac0-23361a42c852" (UID: "66471fb0-5c91-4a44-aac0-23361a42c852"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:33:49.603895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:49.598623    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66471fb0-5c91-4a44-aac0-23361a42c852-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:33:49.603895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:49.598662    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66471fb0-5c91-4a44-aac0-23361a42c852-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:33:49.603895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:49.598678    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jrzz\" (UniqueName: \"kubernetes.io/projected/66471fb0-5c91-4a44-aac0-23361a42c852-kube-api-access-2jrzz\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:33:49.762190 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:49.762124    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66471fb0-5c91-4a44-aac0-23361a42c852-maas-api-tls" pod "66471fb0-5c91-4a44-aac0-23361a42c852" (UID: "66471fb0-5c91-4a44-aac0-23361a42c852"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:33:49.788460 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:49.772861    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66471fb0-5c91-4a44-aac0-23361a42c852-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:33:50.029837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:50.026440    4559 generic.go:356] "Generic (PLEG): container finished" podID="66471fb0-5c91-4a44-aac0-23361a42c852" containerID="c3de853ae45134e2aff1dc84e20e41a2f0fe85c7ff90736bb26d6ca20fcd1c61" exitCode=0
Oct 29 22:33:50.029837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:50.026767    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf" event={"ID":"66471fb0-5c91-4a44-aac0-23361a42c852","Type":"ContainerDied","Data":"c3de853ae45134e2aff1dc84e20e41a2f0fe85c7ff90736bb26d6ca20fcd1c61"}
Oct 29 22:33:50.029837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:50.026828    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"
Oct 29 22:33:50.029837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:50.026911    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf" event={"ID":"66471fb0-5c91-4a44-aac0-23361a42c852","Type":"ContainerDied","Data":"faa04fde41b39bc028db8be5eeb4bd5a384909f36900eed1c9ae99c95c79d3c5"}
Oct 29 22:33:50.029837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:50.026954    4559 scope.go:122] "RemoveContainer" containerID="c3de853ae45134e2aff1dc84e20e41a2f0fe85c7ff90736bb26d6ca20fcd1c61"
Oct 29 22:33:50.364697 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:50.356928    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"]
Oct 29 22:33:50.366694 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:50.365194    4559 scope.go:122] "RemoveContainer" containerID="c3de853ae45134e2aff1dc84e20e41a2f0fe85c7ff90736bb26d6ca20fcd1c61"
Oct 29 22:33:50.366694 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:33:50.365619    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3de853ae45134e2aff1dc84e20e41a2f0fe85c7ff90736bb26d6ca20fcd1c61\": container with ID starting with c3de853ae45134e2aff1dc84e20e41a2f0fe85c7ff90736bb26d6ca20fcd1c61 not found: ID does not exist" containerID="c3de853ae45134e2aff1dc84e20e41a2f0fe85c7ff90736bb26d6ca20fcd1c61"
Oct 29 22:33:50.366694 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:50.365653    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3de853ae45134e2aff1dc84e20e41a2f0fe85c7ff90736bb26d6ca20fcd1c61"} err="failed to get container status \"c3de853ae45134e2aff1dc84e20e41a2f0fe85c7ff90736bb26d6ca20fcd1c61\": rpc error: code = NotFound desc = could not find container \"c3de853ae45134e2aff1dc84e20e41a2f0fe85c7ff90736bb26d6ca20fcd1c61\": container with ID starting with c3de853ae45134e2aff1dc84e20e41a2f0fe85c7ff90736bb26d6ca20fcd1c61 not found: ID does not exist"
Oct 29 22:33:50.366694 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:50.365948    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7dea32d-8458c5b6f4-vvhwf"]
Oct 29 22:33:51.153710 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.153665    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"]
Oct 29 22:33:51.154757 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.154734    4559 cpu_manager.go:414] "containerMap: removing container" podUID="66471fb0-5c91-4a44-aac0-23361a42c852" containerName="maas-api"
Oct 29 22:33:51.155416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.154760    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66471fb0-5c91-4a44-aac0-23361a42c852" containerName="maas-api"
Oct 29 22:33:51.155416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.154930    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="66471fb0-5c91-4a44-aac0-23361a42c852" containerName="maas-api"
Oct 29 22:33:51.160505 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.160483    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.194214 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.194106    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e385da9b-gw-openshift-default-dockercfg-qbqxh\""
Oct 29 22:33:51.196160 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.196124    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"]
Oct 29 22:33:51.230731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.230695    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66471fb0-5c91-4a44-aac0-23361a42c852" path="/var/lib/kubelet/pods/66471fb0-5c91-4a44-aac0-23361a42c852/volumes"
Oct 29 22:33:51.280247 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.276853    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjkz7\" (UniqueName: \"kubernetes.io/projected/0400ea45-bb33-492a-9be4-870fe00e0dcf-kube-api-access-tjkz7\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.280247 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.276922    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-workload-socket\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.280247 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.276965    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-credential-socket\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.280247 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.277032    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0400ea45-bb33-492a-9be4-870fe00e0dcf-istiod-ca-cert\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.280247 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.277099    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-data\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.280247 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.277154    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-token\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.280247 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.277263    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-podinfo\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.280247 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.277320    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-workload-certs\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.280247 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.277386    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-envoy\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.407034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.405163    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-envoy\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.407034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.405309    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjkz7\" (UniqueName: \"kubernetes.io/projected/0400ea45-bb33-492a-9be4-870fe00e0dcf-kube-api-access-tjkz7\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.407034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.405354    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-workload-socket\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.407034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.405389    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-credential-socket\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.407034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.405453    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0400ea45-bb33-492a-9be4-870fe00e0dcf-istiod-ca-cert\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.407034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.405522    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-data\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.407034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.405578    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-token\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.407034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.405684    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-podinfo\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.407034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.405741    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-workload-certs\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.407034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.406270    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-workload-certs\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.489835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.458801    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-envoy\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.489835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.472323    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-workload-socket\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.489835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.472534    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-credential-socket\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.489835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.473155    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0400ea45-bb33-492a-9be4-870fe00e0dcf-istiod-ca-cert\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.489835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.473372    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-data\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.489835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.489700    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-podinfo\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.514835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.510486    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-token\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.519796 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.519669    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjkz7\" (UniqueName: \"kubernetes.io/projected/0400ea45-bb33-492a-9be4-870fe00e0dcf-kube-api-access-tjkz7\") pod \"e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") " pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:51.803538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:51.803426    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:52.211733 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:52.211699    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"]
Oct 29 22:33:52.212480 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:33:52.212454    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0400ea45_bb33_492a_9be4_870fe00e0dcf.slice/crio-1aafa1bed2367951855b13dff576e0f88121cd61ee696cfed9d5a39ebb5df434 WatchSource:0}: Error finding container 1aafa1bed2367951855b13dff576e0f88121cd61ee696cfed9d5a39ebb5df434: Status 404 returned error can't find the container with id 1aafa1bed2367951855b13dff576e0f88121cd61ee696cfed9d5a39ebb5df434
Oct 29 22:33:52.215202 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:52.215162    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:33:52.215276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:52.215234    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:33:52.215276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:52.215269    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:33:53.056772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:53.056737    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv" event={"ID":"0400ea45-bb33-492a-9be4-870fe00e0dcf","Type":"ContainerStarted","Data":"1734268123b155968b7ad947c04035cd416b1f06e04554f64d4ec2f2af1bc9b5"}
Oct 29 22:33:53.056986 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:53.056792    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv" event={"ID":"0400ea45-bb33-492a-9be4-870fe00e0dcf","Type":"ContainerStarted","Data":"1aafa1bed2367951855b13dff576e0f88121cd61ee696cfed9d5a39ebb5df434"}
Oct 29 22:33:53.120317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:53.120263    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv" podStartSLOduration=2.120249109 podStartE2EDuration="2.120249109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:53.118066029 +0000 UTC m=+1172.029509583" watchObservedRunningTime="2026-10-29 22:33:53.120249109 +0000 UTC m=+1172.031692719"
Oct 29 22:33:53.186701 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:53.186663    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:33:53.719824 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:53.719782    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"]
Oct 29 22:33:53.804441 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:53.804244    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:53.810803 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:53.810773    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:54.132120 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:54.132029    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:54.141110 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:54.141077    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:33:58.721032 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:58.720972    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf" podUID="dc75ea97-6e3a-4c05-aad3-9e83e5cf240d" containerName="payload-processing" containerID="cri-o://d4029bf3bb2c343617a8e84cffea9124e159d5da360229cbfa22a2da2ec4d9cd" gracePeriod=25
Oct 29 22:33:59.227438 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:59.227411    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"
Oct 29 22:33:59.359472 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:59.359433    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc75ea97-6e3a-4c05-aad3-9e83e5cf240d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc75ea97-6e3a-4c05-aad3-9e83e5cf240d-plugins-config-volume\") pod \"dc75ea97-6e3a-4c05-aad3-9e83e5cf240d\" (UID: \"dc75ea97-6e3a-4c05-aad3-9e83e5cf240d\") "
Oct 29 22:33:59.359678 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:59.359558    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc75ea97-6e3a-4c05-aad3-9e83e5cf240d-kube-api-access-m2ch4\" (UniqueName: \"kubernetes.io/projected/dc75ea97-6e3a-4c05-aad3-9e83e5cf240d-kube-api-access-m2ch4\") pod \"dc75ea97-6e3a-4c05-aad3-9e83e5cf240d\" (UID: \"dc75ea97-6e3a-4c05-aad3-9e83e5cf240d\") "
Oct 29 22:33:59.359956 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:59.359926    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc75ea97-6e3a-4c05-aad3-9e83e5cf240d-plugins-config-volume" pod "dc75ea97-6e3a-4c05-aad3-9e83e5cf240d" (UID: "dc75ea97-6e3a-4c05-aad3-9e83e5cf240d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:33:59.360234 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:59.360215    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc75ea97-6e3a-4c05-aad3-9e83e5cf240d-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:33:59.362353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:59.362311    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc75ea97-6e3a-4c05-aad3-9e83e5cf240d-kube-api-access-m2ch4" pod "dc75ea97-6e3a-4c05-aad3-9e83e5cf240d" (UID: "dc75ea97-6e3a-4c05-aad3-9e83e5cf240d"). InnerVolumeSpecName "kube-api-access-m2ch4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:33:59.421309 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:59.421267    4559 generic.go:356] "Generic (PLEG): container finished" podID="dc75ea97-6e3a-4c05-aad3-9e83e5cf240d" containerID="d4029bf3bb2c343617a8e84cffea9124e159d5da360229cbfa22a2da2ec4d9cd" exitCode=0
Oct 29 22:33:59.421512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:59.421334    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"
Oct 29 22:33:59.421512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:59.421355    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf" event={"ID":"dc75ea97-6e3a-4c05-aad3-9e83e5cf240d","Type":"ContainerDied","Data":"d4029bf3bb2c343617a8e84cffea9124e159d5da360229cbfa22a2da2ec4d9cd"}
Oct 29 22:33:59.421512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:59.421445    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf" event={"ID":"dc75ea97-6e3a-4c05-aad3-9e83e5cf240d","Type":"ContainerDied","Data":"1f0647d047b82a753c2ec2221fec5666b54f5e2b23d47d97ccf2dcef3d7544a6"}
Oct 29 22:33:59.421512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:59.421473    4559 scope.go:122] "RemoveContainer" containerID="d4029bf3bb2c343617a8e84cffea9124e159d5da360229cbfa22a2da2ec4d9cd"
Oct 29 22:33:59.446523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:59.446192    4559 scope.go:122] "RemoveContainer" containerID="d4029bf3bb2c343617a8e84cffea9124e159d5da360229cbfa22a2da2ec4d9cd"
Oct 29 22:33:59.446866 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:33:59.446751    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4029bf3bb2c343617a8e84cffea9124e159d5da360229cbfa22a2da2ec4d9cd\": container with ID starting with d4029bf3bb2c343617a8e84cffea9124e159d5da360229cbfa22a2da2ec4d9cd not found: ID does not exist" containerID="d4029bf3bb2c343617a8e84cffea9124e159d5da360229cbfa22a2da2ec4d9cd"
Oct 29 22:33:59.446866 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:59.446791    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4029bf3bb2c343617a8e84cffea9124e159d5da360229cbfa22a2da2ec4d9cd"} err="failed to get container status \"d4029bf3bb2c343617a8e84cffea9124e159d5da360229cbfa22a2da2ec4d9cd\": rpc error: code = NotFound desc = could not find container \"d4029bf3bb2c343617a8e84cffea9124e159d5da360229cbfa22a2da2ec4d9cd\": container with ID starting with d4029bf3bb2c343617a8e84cffea9124e159d5da360229cbfa22a2da2ec4d9cd not found: ID does not exist"
Oct 29 22:33:59.462075 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:59.462040    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2ch4\" (UniqueName: \"kubernetes.io/projected/dc75ea97-6e3a-4c05-aad3-9e83e5cf240d-kube-api-access-m2ch4\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:33:59.484650 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:59.484610    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"]
Oct 29 22:33:59.489914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:33:59.489884    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc369ed0-5b7fd658fd-pj5gf"]
Oct 29 22:34:01.230410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:01.230374    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc75ea97-6e3a-4c05-aad3-9e83e5cf240d" path="/var/lib/kubelet/pods/dc75ea97-6e3a-4c05-aad3-9e83e5cf240d/volumes"
Oct 29 22:34:02.031119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:02.031082    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"]
Oct 29 22:34:02.044127 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:02.044096    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38658d1d-npl2p"]
Oct 29 22:34:03.228638 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:03.228605    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec345887-2f41-4a39-ab33-6f3227339294" path="/var/lib/kubelet/pods/ec345887-2f41-4a39-ab33-6f3227339294/volumes"
Oct 29 22:34:06.510510 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:06.510473    4559 generic.go:356] "Generic (PLEG): container finished" podID="10c127bb-5410-4a3c-974b-0b7f4fb31ccb" containerID="640249050da05295c9b701c937b530af139c560d31b271fadc8d42fd7af979b1" exitCode=0
Oct 29 22:34:06.511115 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:06.510544    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw" event={"ID":"10c127bb-5410-4a3c-974b-0b7f4fb31ccb","Type":"ContainerDied","Data":"640249050da05295c9b701c937b530af139c560d31b271fadc8d42fd7af979b1"}
Oct 29 22:34:06.511115 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:06.511027    4559 scope.go:122] "RemoveContainer" containerID="640249050da05295c9b701c937b530af139c560d31b271fadc8d42fd7af979b1"
Oct 29 22:34:06.512738 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:06.512711    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:34:06.512822 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:06.512784    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:34:06.512867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:06.512832    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:34:06.622238 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:06.622201    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"]
Oct 29 22:34:06.622492 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:06.622438    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg" podUID="076c0bce-da05-40aa-a50f-ce3ca7b5e6c9" containerName="maas-api" containerID="cri-o://008db6eea3bb936fb2b98e8acd6701e2f4614ba5f3b0ccdecd3852e8e7428041" gracePeriod=30
Oct 29 22:34:07.111150 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.101038    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"]
Oct 29 22:34:07.279837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.276552    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:34:07.311835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.306159    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71962013-6l9m4"]
Oct 29 22:34:07.329838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.322488    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-openshift-service-ca\") pod \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\" (UID: \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\") "
Oct 29 22:34:07.329838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.322571    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-metrics-certs\") pod \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\" (UID: \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\") "
Oct 29 22:34:07.329838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.322945    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-maas-api-tls\") pod \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\" (UID: \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\") "
Oct 29 22:34:07.329838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.323162    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-kube-api-access-8x2zs\" (UniqueName: \"kubernetes.io/projected/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-kube-api-access-8x2zs\") pod \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\" (UID: \"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9\") "
Oct 29 22:34:07.329838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.326958    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-openshift-service-ca" pod "076c0bce-da05-40aa-a50f-ce3ca7b5e6c9" (UID: "076c0bce-da05-40aa-a50f-ce3ca7b5e6c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:34:07.353919 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.344676    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-kube-api-access-8x2zs" pod "076c0bce-da05-40aa-a50f-ce3ca7b5e6c9" (UID: "076c0bce-da05-40aa-a50f-ce3ca7b5e6c9"). InnerVolumeSpecName "kube-api-access-8x2zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:34:07.386836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.380423    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-maas-api-tls" pod "076c0bce-da05-40aa-a50f-ce3ca7b5e6c9" (UID: "076c0bce-da05-40aa-a50f-ce3ca7b5e6c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:34:07.386836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.380948    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-metrics-certs" pod "076c0bce-da05-40aa-a50f-ce3ca7b5e6c9" (UID: "076c0bce-da05-40aa-a50f-ce3ca7b5e6c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:34:07.436836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.429962    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x2zs\" (UniqueName: \"kubernetes.io/projected/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-kube-api-access-8x2zs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:07.436836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.429998    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:07.436836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.430013    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:07.436836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.430026    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:07.560839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.557455    4559 generic.go:356] "Generic (PLEG): container finished" podID="076c0bce-da05-40aa-a50f-ce3ca7b5e6c9" containerID="008db6eea3bb936fb2b98e8acd6701e2f4614ba5f3b0ccdecd3852e8e7428041" exitCode=0
Oct 29 22:34:07.560839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.557597    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg" event={"ID":"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9","Type":"ContainerDied","Data":"008db6eea3bb936fb2b98e8acd6701e2f4614ba5f3b0ccdecd3852e8e7428041"}
Oct 29 22:34:07.560839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.557652    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg" event={"ID":"076c0bce-da05-40aa-a50f-ce3ca7b5e6c9","Type":"ContainerDied","Data":"b781fc81d9882af36e3a9d78b746a8de7fabcec66ab2bc1b4527740e7911608e"}
Oct 29 22:34:07.560839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.557683    4559 scope.go:122] "RemoveContainer" containerID="008db6eea3bb936fb2b98e8acd6701e2f4614ba5f3b0ccdecd3852e8e7428041"
Oct 29 22:34:07.560839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.557854    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"
Oct 29 22:34:07.719046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.719024    4559 scope.go:122] "RemoveContainer" containerID="008db6eea3bb936fb2b98e8acd6701e2f4614ba5f3b0ccdecd3852e8e7428041"
Oct 29 22:34:07.728133 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:34:07.725691    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"008db6eea3bb936fb2b98e8acd6701e2f4614ba5f3b0ccdecd3852e8e7428041\": container with ID starting with 008db6eea3bb936fb2b98e8acd6701e2f4614ba5f3b0ccdecd3852e8e7428041 not found: ID does not exist" containerID="008db6eea3bb936fb2b98e8acd6701e2f4614ba5f3b0ccdecd3852e8e7428041"
Oct 29 22:34:07.728133 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.725762    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"008db6eea3bb936fb2b98e8acd6701e2f4614ba5f3b0ccdecd3852e8e7428041"} err="failed to get container status \"008db6eea3bb936fb2b98e8acd6701e2f4614ba5f3b0ccdecd3852e8e7428041\": rpc error: code = NotFound desc = could not find container \"008db6eea3bb936fb2b98e8acd6701e2f4614ba5f3b0ccdecd3852e8e7428041\": container with ID starting with 008db6eea3bb936fb2b98e8acd6701e2f4614ba5f3b0ccdecd3852e8e7428041 not found: ID does not exist"
Oct 29 22:34:07.728133 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.726689    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw" event={"ID":"10c127bb-5410-4a3c-974b-0b7f4fb31ccb","Type":"ContainerStarted","Data":"ac495f2d59a3e3d66402707daffa5133097d9088278021fc56bbdf5abc9151a5"}
Oct 29 22:34:07.769834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.769721    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"]
Oct 29 22:34:07.781799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:07.781715    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc369ed0-ddd8d9574-ldwgg"]
Oct 29 22:34:08.348609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:08.344186    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:34:08.348981 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:08.348954    4559 patch_prober.go:29] interesting pod/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:34:08.349163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:08.349135    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw" podUID="10c127bb-5410-4a3c-974b-0b7f4fb31ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 29 22:34:08.882255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:08.882212    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"]
Oct 29 22:34:09.342530 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.332302    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="076c0bce-da05-40aa-a50f-ce3ca7b5e6c9" path="/var/lib/kubelet/pods/076c0bce-da05-40aa-a50f-ce3ca7b5e6c9/volumes"
Oct 29 22:34:09.345486 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.345440    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="806d27e8-e0bd-4f8a-ab8e-067352d7f75f" path="/var/lib/kubelet/pods/806d27e8-e0bd-4f8a-ab8e-067352d7f75f/volumes"
Oct 29 22:34:09.363018 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.362986    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"]
Oct 29 22:34:09.364536 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.364502    4559 cpu_manager.go:414] "containerMap: removing container" podUID="dc75ea97-6e3a-4c05-aad3-9e83e5cf240d" containerName="payload-processing"
Oct 29 22:34:09.364712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.364696    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc75ea97-6e3a-4c05-aad3-9e83e5cf240d" containerName="payload-processing"
Oct 29 22:34:09.365082 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.365050    4559 cpu_manager.go:414] "containerMap: removing container" podUID="076c0bce-da05-40aa-a50f-ce3ca7b5e6c9" containerName="maas-api"
Oct 29 22:34:09.365223 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.365210    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076c0bce-da05-40aa-a50f-ce3ca7b5e6c9" containerName="maas-api"
Oct 29 22:34:09.365566 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.365552    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc75ea97-6e3a-4c05-aad3-9e83e5cf240d" containerName="payload-processing"
Oct 29 22:34:09.365692 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.365681    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="076c0bce-da05-40aa-a50f-ce3ca7b5e6c9" containerName="maas-api"
Oct 29 22:34:09.412859 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.398385    4559 patch_prober.go:29] interesting pod/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:34:09.412859 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.398449    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw" podUID="10c127bb-5410-4a3c-974b-0b7f4fb31ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 29 22:34:09.429331 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.428122    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"]
Oct 29 22:34:09.429331 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.428262    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:09.449756 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.449701    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e385da9b-serving-cert\""
Oct 29 22:34:09.515375 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.515287    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-maas-api-tls\") pod \"maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf\" (UID: \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:09.515720 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.515684    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-metrics-certs\") pod \"maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf\" (UID: \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:09.515986 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.515965    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-openshift-service-ca\") pod \"maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf\" (UID: \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:09.516147 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.516132    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f4g7\" (UniqueName: \"kubernetes.io/projected/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-kube-api-access-9f4g7\") pod \"maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf\" (UID: \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:09.624929 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.624875    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-maas-api-tls\") pod \"maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf\" (UID: \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:09.637230 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.637193    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-metrics-certs\") pod \"maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf\" (UID: \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:09.637420 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.637338    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-openshift-service-ca\") pod \"maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf\" (UID: \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:09.637420 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.637411    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f4g7\" (UniqueName: \"kubernetes.io/projected/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-kube-api-access-9f4g7\") pod \"maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf\" (UID: \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:09.674274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.662549    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-maas-api-tls\") pod \"maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf\" (UID: \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:09.674274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.663372    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-openshift-service-ca\") pod \"maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf\" (UID: \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:09.706966 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.677504    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-metrics-certs\") pod \"maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf\" (UID: \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:09.786021 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.785770    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f4g7\" (UniqueName: \"kubernetes.io/projected/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-kube-api-access-9f4g7\") pod \"maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf\" (UID: \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:09.791658 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:09.791614    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw" podUID="10c127bb-5410-4a3c-974b-0b7f4fb31ccb" containerName="istio-proxy" containerID="cri-o://ac495f2d59a3e3d66402707daffa5133097d9088278021fc56bbdf5abc9151a5" gracePeriod=30
Oct 29 22:34:10.057793 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:10.057757    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:10.504019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:10.503984    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"]
Oct 29 22:34:10.928845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:10.925733    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf" event={"ID":"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf","Type":"ContainerStarted","Data":"ff3316de202d59162e38b9b7bae3f58c106ac0a90fe3948069783025ab663431"}
Oct 29 22:34:11.935879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:11.935834    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf" event={"ID":"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf","Type":"ContainerStarted","Data":"57be6156401288d5ae1a00ecbedd7fe55514592667a929ad01ee6953cda2df4a"}
Oct 29 22:34:11.937874 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:11.937849    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:11.992954 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:11.992899    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf" podStartSLOduration=2.501712401 podStartE2EDuration="2.992885894s" totalImagesPullingTime="491.173493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:11.98974597 +0000 UTC m=+1190.901189548" watchObservedRunningTime="2026-10-29 22:34:11.992885894 +0000 UTC m=+1190.904329490"
Oct 29 22:34:13.696324 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:34:13.696282    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:34:15.049925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.049888    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"]
Oct 29 22:34:15.054402 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.054377    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"
Oct 29 22:34:15.076396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.076361    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"]
Oct 29 22:34:15.085751 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.084545    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-6c6e65e5\""
Oct 29 22:34:15.085751 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.084686    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-6c6e65e5-dockercfg-bvk5d\""
Oct 29 22:34:15.386453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.384324    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz66w\" (UniqueName: \"kubernetes.io/projected/bcf9d46d-ec5f-407c-bb46-563f65581bed-kube-api-access-xz66w\") pod \"payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn\" (UID: \"bcf9d46d-ec5f-407c-bb46-563f65581bed\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"
Oct 29 22:34:15.386453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.384472    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcf9d46d-ec5f-407c-bb46-563f65581bed-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn\" (UID: \"bcf9d46d-ec5f-407c-bb46-563f65581bed\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"
Oct 29 22:34:15.455849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.455070    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"]
Oct 29 22:34:15.466947 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.463006    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"]
Oct 29 22:34:15.466947 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.463129    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"
Oct 29 22:34:15.485984 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.485943    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcf9d46d-ec5f-407c-bb46-563f65581bed-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn\" (UID: \"bcf9d46d-ec5f-407c-bb46-563f65581bed\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"
Oct 29 22:34:15.486259 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.486223    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz66w\" (UniqueName: \"kubernetes.io/projected/bcf9d46d-ec5f-407c-bb46-563f65581bed-kube-api-access-xz66w\") pod \"payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn\" (UID: \"bcf9d46d-ec5f-407c-bb46-563f65581bed\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"
Oct 29 22:34:15.487013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.486983    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcf9d46d-ec5f-407c-bb46-563f65581bed-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn\" (UID: \"bcf9d46d-ec5f-407c-bb46-563f65581bed\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"
Oct 29 22:34:15.508051 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.508017    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz66w\" (UniqueName: \"kubernetes.io/projected/bcf9d46d-ec5f-407c-bb46-563f65581bed-kube-api-access-xz66w\") pod \"payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn\" (UID: \"bcf9d46d-ec5f-407c-bb46-563f65581bed\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"
Oct 29 22:34:15.587495 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.587457    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72172fd1-33a9-4322-9d65-e923277c3b51-plugins-config-volume\") pod \"payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh\" (UID: \"72172fd1-33a9-4322-9d65-e923277c3b51\") " pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"
Oct 29 22:34:15.587871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.587800    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkldp\" (UniqueName: \"kubernetes.io/projected/72172fd1-33a9-4322-9d65-e923277c3b51-kube-api-access-lkldp\") pod \"payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh\" (UID: \"72172fd1-33a9-4322-9d65-e923277c3b51\") " pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"
Oct 29 22:34:15.671338 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.670085    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"
Oct 29 22:34:15.695572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.695521    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkldp\" (UniqueName: \"kubernetes.io/projected/72172fd1-33a9-4322-9d65-e923277c3b51-kube-api-access-lkldp\") pod \"payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh\" (UID: \"72172fd1-33a9-4322-9d65-e923277c3b51\") " pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"
Oct 29 22:34:15.697160 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.695717    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72172fd1-33a9-4322-9d65-e923277c3b51-plugins-config-volume\") pod \"payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh\" (UID: \"72172fd1-33a9-4322-9d65-e923277c3b51\") " pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"
Oct 29 22:34:15.697160 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.697104    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72172fd1-33a9-4322-9d65-e923277c3b51-plugins-config-volume\") pod \"payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh\" (UID: \"72172fd1-33a9-4322-9d65-e923277c3b51\") " pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"
Oct 29 22:34:15.741890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.741846    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkldp\" (UniqueName: \"kubernetes.io/projected/72172fd1-33a9-4322-9d65-e923277c3b51-kube-api-access-lkldp\") pod \"payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh\" (UID: \"72172fd1-33a9-4322-9d65-e923277c3b51\") " pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"
Oct 29 22:34:15.783864 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.782896    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"
Oct 29 22:34:15.847877 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:15.845432    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:34:16.061905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:16.061868    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"]
Oct 29 22:34:16.062324 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:34:16.061927    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf9d46d_ec5f_407c_bb46_563f65581bed.slice/crio-a746c810001a55d8a1ccca7201d5d1d21ae002482bdc14902b02345c0aa966f4 WatchSource:0}: Error finding container a746c810001a55d8a1ccca7201d5d1d21ae002482bdc14902b02345c0aa966f4: Status 404 returned error can't find the container with id a746c810001a55d8a1ccca7201d5d1d21ae002482bdc14902b02345c0aa966f4
Oct 29 22:34:16.113348 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:34:16.113315    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72172fd1_33a9_4322_9d65_e923277c3b51.slice/crio-5ce6100fcc69ac31f811a92ff43984d8840f9e14a12c4c7722d833a8ff83685a WatchSource:0}: Error finding container 5ce6100fcc69ac31f811a92ff43984d8840f9e14a12c4c7722d833a8ff83685a: Status 404 returned error can't find the container with id 5ce6100fcc69ac31f811a92ff43984d8840f9e14a12c4c7722d833a8ff83685a
Oct 29 22:34:16.114607 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:16.114581    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"]
Oct 29 22:34:17.001435 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:17.001393    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn" event={"ID":"bcf9d46d-ec5f-407c-bb46-563f65581bed","Type":"ContainerStarted","Data":"2057b14fc7532a9f8825999e44fb4baac8dcd5d8987a117702942bd907c0a37b"}
Oct 29 22:34:17.001660 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:17.001572    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"
Oct 29 22:34:17.001660 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:17.001617    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn" event={"ID":"bcf9d46d-ec5f-407c-bb46-563f65581bed","Type":"ContainerStarted","Data":"a746c810001a55d8a1ccca7201d5d1d21ae002482bdc14902b02345c0aa966f4"}
Oct 29 22:34:17.003373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:17.003345    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh" event={"ID":"72172fd1-33a9-4322-9d65-e923277c3b51","Type":"ContainerStarted","Data":"b8602a982d0ece76f0a1d02c03854735b00c7ea752120db45743e936a908eb7f"}
Oct 29 22:34:17.003542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:17.003396    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh" event={"ID":"72172fd1-33a9-4322-9d65-e923277c3b51","Type":"ContainerStarted","Data":"5ce6100fcc69ac31f811a92ff43984d8840f9e14a12c4c7722d833a8ff83685a"}
Oct 29 22:34:17.003542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:17.003439    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"
Oct 29 22:34:17.043875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:17.043787    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn" podStartSLOduration=2.043771849 podStartE2EDuration="2.043771849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:17.041960194 +0000 UTC m=+1195.953403747" watchObservedRunningTime="2026-10-29 22:34:17.043771849 +0000 UTC m=+1195.955215403"
Oct 29 22:34:17.082477 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:17.082422    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh" podStartSLOduration=2.082406643 podStartE2EDuration="2.082406643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:17.081916427 +0000 UTC m=+1195.993359995" watchObservedRunningTime="2026-10-29 22:34:17.082406643 +0000 UTC m=+1195.993850196"
Oct 29 22:34:18.984375 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:18.984343    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:19.321398 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.321368    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:34:19.481967 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.481929    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-credential-socket\") pod \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") "
Oct 29 22:34:19.482148 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.482019    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-data\") pod \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") "
Oct 29 22:34:19.482148 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.482112    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-kube-api-access-qtrrd\" (UniqueName: \"kubernetes.io/projected/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-kube-api-access-qtrrd\") pod \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") "
Oct 29 22:34:19.482148 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.482142    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istiod-ca-cert\") pod \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") "
Oct 29 22:34:19.482321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.482217    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-workload-certs\") pod \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") "
Oct 29 22:34:19.482321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.482257    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-credential-socket" pod "10c127bb-5410-4a3c-974b-0b7f4fb31ccb" (UID: "10c127bb-5410-4a3c-974b-0b7f4fb31ccb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:34:19.482321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.482267    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-data" pod "10c127bb-5410-4a3c-974b-0b7f4fb31ccb" (UID: "10c127bb-5410-4a3c-974b-0b7f4fb31ccb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:34:19.482321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.482306    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-workload-socket\") pod \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") "
Oct 29 22:34:19.482509 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.482333    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-envoy\") pod \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") "
Oct 29 22:34:19.482509 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.482444    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-token\" (UniqueName: \"kubernetes.io/projected/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-token\") pod \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") "
Oct 29 22:34:19.482908 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.482638    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-podinfo\") pod \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\" (UID: \"10c127bb-5410-4a3c-974b-0b7f4fb31ccb\") "
Oct 29 22:34:19.482908 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.482655    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-workload-socket" pod "10c127bb-5410-4a3c-974b-0b7f4fb31ccb" (UID: "10c127bb-5410-4a3c-974b-0b7f4fb31ccb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:34:19.482908 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.482722    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-workload-certs" pod "10c127bb-5410-4a3c-974b-0b7f4fb31ccb" (UID: "10c127bb-5410-4a3c-974b-0b7f4fb31ccb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:34:19.482908 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.482744    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istiod-ca-cert" pod "10c127bb-5410-4a3c-974b-0b7f4fb31ccb" (UID: "10c127bb-5410-4a3c-974b-0b7f4fb31ccb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:34:19.483942 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.483917    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:19.483942 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.483943    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:19.484110 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.483956    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:19.484110 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.483968    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:19.484110 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.483980    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:19.486210 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.485900    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-envoy" pod "10c127bb-5410-4a3c-974b-0b7f4fb31ccb" (UID: "10c127bb-5410-4a3c-974b-0b7f4fb31ccb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:34:19.486210 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.486020    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-token" pod "10c127bb-5410-4a3c-974b-0b7f4fb31ccb" (UID: "10c127bb-5410-4a3c-974b-0b7f4fb31ccb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:34:19.486210 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.486037    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-kube-api-access-qtrrd" pod "10c127bb-5410-4a3c-974b-0b7f4fb31ccb" (UID: "10c127bb-5410-4a3c-974b-0b7f4fb31ccb"). InnerVolumeSpecName "kube-api-access-qtrrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:34:19.486442 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.486216    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-podinfo" pod "10c127bb-5410-4a3c-974b-0b7f4fb31ccb" (UID: "10c127bb-5410-4a3c-974b-0b7f4fb31ccb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:34:19.585544 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.585452    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:19.585544 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.585490    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:19.585544 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.585500    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:19.585544 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:19.585509    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtrrd\" (UniqueName: \"kubernetes.io/projected/10c127bb-5410-4a3c-974b-0b7f4fb31ccb-kube-api-access-qtrrd\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:20.099839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:20.092338    4559 generic.go:356] "Generic (PLEG): container finished" podID="10c127bb-5410-4a3c-974b-0b7f4fb31ccb" containerID="ac495f2d59a3e3d66402707daffa5133097d9088278021fc56bbdf5abc9151a5" exitCode=0
Oct 29 22:34:20.099839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:20.092666    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"
Oct 29 22:34:20.099839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:20.097027    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw" event={"ID":"10c127bb-5410-4a3c-974b-0b7f4fb31ccb","Type":"ContainerDied","Data":"ac495f2d59a3e3d66402707daffa5133097d9088278021fc56bbdf5abc9151a5"}
Oct 29 22:34:20.099839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:20.097155    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw" event={"ID":"10c127bb-5410-4a3c-974b-0b7f4fb31ccb","Type":"ContainerDied","Data":"d9afa90759a55b933420bd772b178ce40d6df4ce07f57fcb997a640d1101e4cb"}
Oct 29 22:34:20.099839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:20.097215    4559 scope.go:122] "RemoveContainer" containerID="ac495f2d59a3e3d66402707daffa5133097d9088278021fc56bbdf5abc9151a5"
Oct 29 22:34:20.130909 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:20.130860    4559 scope.go:122] "RemoveContainer" containerID="640249050da05295c9b701c937b530af139c560d31b271fadc8d42fd7af979b1"
Oct 29 22:34:20.143869 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:20.143838    4559 scope.go:122] "RemoveContainer" containerID="ac495f2d59a3e3d66402707daffa5133097d9088278021fc56bbdf5abc9151a5"
Oct 29 22:34:20.144264 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:34:20.144243    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac495f2d59a3e3d66402707daffa5133097d9088278021fc56bbdf5abc9151a5\": container with ID starting with ac495f2d59a3e3d66402707daffa5133097d9088278021fc56bbdf5abc9151a5 not found: ID does not exist" containerID="ac495f2d59a3e3d66402707daffa5133097d9088278021fc56bbdf5abc9151a5"
Oct 29 22:34:20.144346 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:20.144280    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac495f2d59a3e3d66402707daffa5133097d9088278021fc56bbdf5abc9151a5"} err="failed to get container status \"ac495f2d59a3e3d66402707daffa5133097d9088278021fc56bbdf5abc9151a5\": rpc error: code = NotFound desc = could not find container \"ac495f2d59a3e3d66402707daffa5133097d9088278021fc56bbdf5abc9151a5\": container with ID starting with ac495f2d59a3e3d66402707daffa5133097d9088278021fc56bbdf5abc9151a5 not found: ID does not exist"
Oct 29 22:34:20.144346 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:20.144306    4559 scope.go:122] "RemoveContainer" containerID="640249050da05295c9b701c937b530af139c560d31b271fadc8d42fd7af979b1"
Oct 29 22:34:20.144647 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:34:20.144626    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"640249050da05295c9b701c937b530af139c560d31b271fadc8d42fd7af979b1\": container with ID starting with 640249050da05295c9b701c937b530af139c560d31b271fadc8d42fd7af979b1 not found: ID does not exist" containerID="640249050da05295c9b701c937b530af139c560d31b271fadc8d42fd7af979b1"
Oct 29 22:34:20.144697 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:20.144654    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"640249050da05295c9b701c937b530af139c560d31b271fadc8d42fd7af979b1"} err="failed to get container status \"640249050da05295c9b701c937b530af139c560d31b271fadc8d42fd7af979b1\": rpc error: code = NotFound desc = could not find container \"640249050da05295c9b701c937b530af139c560d31b271fadc8d42fd7af979b1\": container with ID starting with 640249050da05295c9b701c937b530af139c560d31b271fadc8d42fd7af979b1 not found: ID does not exist"
Oct 29 22:34:20.172064 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:20.172029    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"]
Oct 29 22:34:20.182595 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:20.182563    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-dc369ed0-openshift-default-858c684c57-npckw"]
Oct 29 22:34:21.229638 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:21.229606    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10c127bb-5410-4a3c-974b-0b7f4fb31ccb" path="/var/lib/kubelet/pods/10c127bb-5410-4a3c-974b-0b7f4fb31ccb/volumes"
Oct 29 22:34:21.409621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:21.409600    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:34:21.414585 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:21.414560    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:34:28.016652 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.016625    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"
Oct 29 22:34:28.017165 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.016730    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"
Oct 29 22:34:28.033126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.033088    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"]
Oct 29 22:34:28.033889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.033865    4559 cpu_manager.go:414] "containerMap: removing container" podUID="10c127bb-5410-4a3c-974b-0b7f4fb31ccb" containerName="istio-proxy"
Oct 29 22:34:28.033889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.033887    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c127bb-5410-4a3c-974b-0b7f4fb31ccb" containerName="istio-proxy"
Oct 29 22:34:28.034085 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.033909    4559 cpu_manager.go:414] "containerMap: removing container" podUID="10c127bb-5410-4a3c-974b-0b7f4fb31ccb" containerName="istio-proxy"
Oct 29 22:34:28.034085 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.033916    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c127bb-5410-4a3c-974b-0b7f4fb31ccb" containerName="istio-proxy"
Oct 29 22:34:28.034182 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.034096    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c127bb-5410-4a3c-974b-0b7f4fb31ccb" containerName="istio-proxy"
Oct 29 22:34:28.034182 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.034113    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c127bb-5410-4a3c-974b-0b7f4fb31ccb" containerName="istio-proxy"
Oct 29 22:34:28.044452 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.042820    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:34:28.074799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.074746    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"]
Oct 29 22:34:28.086434 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.086408    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-6c6e65e5-serving-cert\""
Oct 29 22:34:28.176126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.176085    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158aa4ff-b698-4c6e-b6c5-a3399296dad5-metrics-certs\") pod \"maas-api-e2e-derive-6c6e65e5-6444467978-7tst5\" (UID: \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:34:28.176313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.176236    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/158aa4ff-b698-4c6e-b6c5-a3399296dad5-openshift-service-ca\") pod \"maas-api-e2e-derive-6c6e65e5-6444467978-7tst5\" (UID: \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:34:28.176382 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.176365    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq8rv\" (UniqueName: \"kubernetes.io/projected/158aa4ff-b698-4c6e-b6c5-a3399296dad5-kube-api-access-pq8rv\") pod \"maas-api-e2e-derive-6c6e65e5-6444467978-7tst5\" (UID: \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:34:28.176468 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.176434    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/158aa4ff-b698-4c6e-b6c5-a3399296dad5-maas-api-tls\") pod \"maas-api-e2e-derive-6c6e65e5-6444467978-7tst5\" (UID: \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:34:28.294886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.285421    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158aa4ff-b698-4c6e-b6c5-a3399296dad5-metrics-certs\") pod \"maas-api-e2e-derive-6c6e65e5-6444467978-7tst5\" (UID: \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:34:28.294886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.285541    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/158aa4ff-b698-4c6e-b6c5-a3399296dad5-openshift-service-ca\") pod \"maas-api-e2e-derive-6c6e65e5-6444467978-7tst5\" (UID: \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:34:28.294886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.285758    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq8rv\" (UniqueName: \"kubernetes.io/projected/158aa4ff-b698-4c6e-b6c5-a3399296dad5-kube-api-access-pq8rv\") pod \"maas-api-e2e-derive-6c6e65e5-6444467978-7tst5\" (UID: \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:34:28.294886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.285861    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/158aa4ff-b698-4c6e-b6c5-a3399296dad5-maas-api-tls\") pod \"maas-api-e2e-derive-6c6e65e5-6444467978-7tst5\" (UID: \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:34:28.294886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.294747    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/158aa4ff-b698-4c6e-b6c5-a3399296dad5-openshift-service-ca\") pod \"maas-api-e2e-derive-6c6e65e5-6444467978-7tst5\" (UID: \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:34:28.359464 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.359428    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158aa4ff-b698-4c6e-b6c5-a3399296dad5-metrics-certs\") pod \"maas-api-e2e-derive-6c6e65e5-6444467978-7tst5\" (UID: \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:34:28.374516 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.374448    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq8rv\" (UniqueName: \"kubernetes.io/projected/158aa4ff-b698-4c6e-b6c5-a3399296dad5-kube-api-access-pq8rv\") pod \"maas-api-e2e-derive-6c6e65e5-6444467978-7tst5\" (UID: \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:34:28.394847 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.391498    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/158aa4ff-b698-4c6e-b6c5-a3399296dad5-maas-api-tls\") pod \"maas-api-e2e-derive-6c6e65e5-6444467978-7tst5\" (UID: \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:34:28.517838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.509069    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:34:28.691924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:28.689902    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:34:29.090435 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:29.090398    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"]
Oct 29 22:34:29.090943 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:34:29.090466    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158aa4ff_b698_4c6e_b6c5_a3399296dad5.slice/crio-f0ad0aa8c791f2278bb204249bdfb7c67b420c249427d2414ea7f36916ec2c42 WatchSource:0}: Error finding container f0ad0aa8c791f2278bb204249bdfb7c67b420c249427d2414ea7f36916ec2c42: Status 404 returned error can't find the container with id f0ad0aa8c791f2278bb204249bdfb7c67b420c249427d2414ea7f36916ec2c42
Oct 29 22:34:29.194545 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:29.194502    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5" event={"ID":"158aa4ff-b698-4c6e-b6c5-a3399296dad5","Type":"ContainerStarted","Data":"f0ad0aa8c791f2278bb204249bdfb7c67b420c249427d2414ea7f36916ec2c42"}
Oct 29 22:34:30.243840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:30.237028    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5" event={"ID":"158aa4ff-b698-4c6e-b6c5-a3399296dad5","Type":"ContainerStarted","Data":"c7777adbb235f3f7deea554e223dcf48aa85d60d74920099b2d5037ab7e1513d"}
Oct 29 22:34:30.243840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:30.238234    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:34:35.390377 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.390263    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5" podStartSLOduration=6.919051995 podStartE2EDuration="7.390241428s" totalImagesPullingTime="471.189433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:30.295574819 +0000 UTC m=+1209.207018384" watchObservedRunningTime="2026-10-29 22:34:35.390241428 +0000 UTC m=+1214.301684980"
Oct 29 22:34:35.393606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.393574    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"]
Oct 29 22:34:35.399672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.399646    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.415053 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.415021    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-06a95f-openshift-default-dockercfg-gb4cw\""
Oct 29 22:34:35.431855 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.431785    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"]
Oct 29 22:34:35.472894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.472548    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-token\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.472894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.472622    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istiod-ca-cert\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.472894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.472660    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-data\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.472894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.472721    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw7hc\" (UniqueName: \"kubernetes.io/projected/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-kube-api-access-bw7hc\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.472894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.472761    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-credential-socket\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.472894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.472858    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-workload-socket\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.473430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.472986    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-workload-certs\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.473430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.473020    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-podinfo\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.473430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.473046    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-envoy\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.636836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.636665    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-podinfo\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.636836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.636732    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-envoy\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.637099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.636936    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-token\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.637099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.636995    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istiod-ca-cert\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.637099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.637043    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-data\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.637259 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.637120    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw7hc\" (UniqueName: \"kubernetes.io/projected/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-kube-api-access-bw7hc\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.637259 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.637164    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-credential-socket\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.637259 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.637237    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-workload-socket\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.637396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.637380    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-workload-certs\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.694920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.637922    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-workload-socket\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.694920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.638204    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-data\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.694920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.638764    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-credential-socket\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.694920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.679305    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-workload-certs\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.709864 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.704731    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istiod-ca-cert\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.709864 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.705431    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-envoy\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.729845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.725728    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-podinfo\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.729845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.728773    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw7hc\" (UniqueName: \"kubernetes.io/projected/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-kube-api-access-bw7hc\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:35.797835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:35.795345    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-token\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:36.028879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:36.028105    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:36.485248 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:34:36.485217    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244c0d1f_3ddb_423b_9cc8_bde3dbd0064c.slice/crio-aa52ad6be59315adaf00034584a3a20eae85010dc6457acf2953c77c40d91354 WatchSource:0}: Error finding container aa52ad6be59315adaf00034584a3a20eae85010dc6457acf2953c77c40d91354: Status 404 returned error can't find the container with id aa52ad6be59315adaf00034584a3a20eae85010dc6457acf2953c77c40d91354
Oct 29 22:34:36.486677 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:36.486649    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"]
Oct 29 22:34:36.488298 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:36.488269    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:34:36.488354 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:36.488335    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:34:36.488394 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:36.488364    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:34:37.362672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:37.362590    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:34:37.408085 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:37.408052    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh" event={"ID":"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c","Type":"ContainerStarted","Data":"ee567f4816a59d997f20a6010840ceabcf7bd1ac8e50bb6739a48263fb292279"}
Oct 29 22:34:37.408264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:37.408115    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh" event={"ID":"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c","Type":"ContainerStarted","Data":"aa52ad6be59315adaf00034584a3a20eae85010dc6457acf2953c77c40d91354"}
Oct 29 22:34:37.474227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:37.474161    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh" podStartSLOduration=2.474137727 podStartE2EDuration="2.474137727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:37.468125527 +0000 UTC m=+1216.379569081" watchObservedRunningTime="2026-10-29 22:34:37.474137727 +0000 UTC m=+1216.385581282"
Oct 29 22:34:38.028639 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:38.028607    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:38.035049 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:38.035019    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:38.417559 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:38.417518    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:38.418529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:38.418510    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:34:40.176835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:40.173971    4559 patch_prober.go:29] interesting pod/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:34:40.176835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:40.174038    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv" podUID="0400ea45-bb33-492a-9be4-870fe00e0dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:34:41.994196 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:41.994159    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"]
Oct 29 22:34:41.995126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:41.995018    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf" podUID="d3e8bf7a-abd1-48ee-aa36-462a9e9385bf" containerName="maas-api" containerID="cri-o://57be6156401288d5ae1a00ecbedd7fe55514592667a929ad01ee6953cda2df4a" gracePeriod=30
Oct 29 22:34:42.347743 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.347707    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"]
Oct 29 22:34:42.348312 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.348289    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:42.348539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.348522    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e8bf7a-abd1-48ee-aa36-462a9e9385bf" containerName="maas-api"
Oct 29 22:34:42.352078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.352060    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"
Oct 29 22:34:42.367958 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.367912    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-06a95f\""
Oct 29 22:34:42.368149 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.368084    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-06a95f-dockercfg-6k7rl\""
Oct 29 22:34:42.374720 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.374689    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"]
Oct 29 22:34:42.431454 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.431293    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-metrics-certs\") pod \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\" (UID: \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\") "
Oct 29 22:34:42.431737 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.431636    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-kube-api-access-9f4g7\" (UniqueName: \"kubernetes.io/projected/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-kube-api-access-9f4g7\") pod \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\" (UID: \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\") "
Oct 29 22:34:42.431737 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.431681    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-openshift-service-ca\") pod \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\" (UID: \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\") "
Oct 29 22:34:42.433005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.431766    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-maas-api-tls\") pod \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\" (UID: \"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf\") "
Oct 29 22:34:42.433005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.432333    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4010ffa9-1ddd-4f51-afc9-eb5ea50bf969-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j\" (UID: \"4010ffa9-1ddd-4f51-afc9-eb5ea50bf969\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"
Oct 29 22:34:42.433005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.432383    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-openshift-service-ca" pod "d3e8bf7a-abd1-48ee-aa36-462a9e9385bf" (UID: "d3e8bf7a-abd1-48ee-aa36-462a9e9385bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:34:42.433005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.432530    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr5ph\" (UniqueName: \"kubernetes.io/projected/4010ffa9-1ddd-4f51-afc9-eb5ea50bf969-kube-api-access-hr5ph\") pod \"payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j\" (UID: \"4010ffa9-1ddd-4f51-afc9-eb5ea50bf969\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"
Oct 29 22:34:42.433005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.432673    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:42.449230 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.449189    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-maas-api-tls" pod "d3e8bf7a-abd1-48ee-aa36-462a9e9385bf" (UID: "d3e8bf7a-abd1-48ee-aa36-462a9e9385bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:34:42.449417 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.449398    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-metrics-certs" pod "d3e8bf7a-abd1-48ee-aa36-462a9e9385bf" (UID: "d3e8bf7a-abd1-48ee-aa36-462a9e9385bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:34:42.449974 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.449722    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-kube-api-access-9f4g7" pod "d3e8bf7a-abd1-48ee-aa36-462a9e9385bf" (UID: "d3e8bf7a-abd1-48ee-aa36-462a9e9385bf"). InnerVolumeSpecName "kube-api-access-9f4g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:34:42.456543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.456508    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"]
Oct 29 22:34:42.458277 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.458250    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d3e8bf7a-abd1-48ee-aa36-462a9e9385bf" containerName="maas-api"
Oct 29 22:34:42.458277 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.458276    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e8bf7a-abd1-48ee-aa36-462a9e9385bf" containerName="maas-api"
Oct 29 22:34:42.463439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.463385    4559 generic.go:356] "Generic (PLEG): container finished" podID="0400ea45-bb33-492a-9be4-870fe00e0dcf" containerID="1734268123b155968b7ad947c04035cd416b1f06e04554f64d4ec2f2af1bc9b5" exitCode=0
Oct 29 22:34:42.474979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.470900    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv" event={"ID":"0400ea45-bb33-492a-9be4-870fe00e0dcf","Type":"ContainerDied","Data":"1734268123b155968b7ad947c04035cd416b1f06e04554f64d4ec2f2af1bc9b5"}
Oct 29 22:34:42.474979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.471603    4559 scope.go:122] "RemoveContainer" containerID="1734268123b155968b7ad947c04035cd416b1f06e04554f64d4ec2f2af1bc9b5"
Oct 29 22:34:42.474979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.471970    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"
Oct 29 22:34:42.476910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.476879    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"]
Oct 29 22:34:42.481582 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.481544    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:34:42.481846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.481786    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:34:42.482018 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.481999    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:34:42.483451 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.483424    4559 generic.go:356] "Generic (PLEG): container finished" podID="d3e8bf7a-abd1-48ee-aa36-462a9e9385bf" containerID="57be6156401288d5ae1a00ecbedd7fe55514592667a929ad01ee6953cda2df4a" exitCode=0
Oct 29 22:34:42.483611 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.483591    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf" event={"ID":"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf","Type":"ContainerDied","Data":"57be6156401288d5ae1a00ecbedd7fe55514592667a929ad01ee6953cda2df4a"}
Oct 29 22:34:42.483672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.483643    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf" event={"ID":"d3e8bf7a-abd1-48ee-aa36-462a9e9385bf","Type":"ContainerDied","Data":"ff3316de202d59162e38b9b7bae3f58c106ac0a90fe3948069783025ab663431"}
Oct 29 22:34:42.483709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.483671    4559 scope.go:122] "RemoveContainer" containerID="57be6156401288d5ae1a00ecbedd7fe55514592667a929ad01ee6953cda2df4a"
Oct 29 22:34:42.483789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.483776    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"
Oct 29 22:34:42.500963 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.500941    4559 scope.go:122] "RemoveContainer" containerID="57be6156401288d5ae1a00ecbedd7fe55514592667a929ad01ee6953cda2df4a"
Oct 29 22:34:42.501286 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:34:42.501265    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57be6156401288d5ae1a00ecbedd7fe55514592667a929ad01ee6953cda2df4a\": container with ID starting with 57be6156401288d5ae1a00ecbedd7fe55514592667a929ad01ee6953cda2df4a not found: ID does not exist" containerID="57be6156401288d5ae1a00ecbedd7fe55514592667a929ad01ee6953cda2df4a"
Oct 29 22:34:42.501384 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.501304    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57be6156401288d5ae1a00ecbedd7fe55514592667a929ad01ee6953cda2df4a"} err="failed to get container status \"57be6156401288d5ae1a00ecbedd7fe55514592667a929ad01ee6953cda2df4a\": rpc error: code = NotFound desc = could not find container \"57be6156401288d5ae1a00ecbedd7fe55514592667a929ad01ee6953cda2df4a\": container with ID starting with 57be6156401288d5ae1a00ecbedd7fe55514592667a929ad01ee6953cda2df4a not found: ID does not exist"
Oct 29 22:34:42.632912 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.617364    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4010ffa9-1ddd-4f51-afc9-eb5ea50bf969-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j\" (UID: \"4010ffa9-1ddd-4f51-afc9-eb5ea50bf969\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"
Oct 29 22:34:42.632912 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.617508    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr5ph\" (UniqueName: \"kubernetes.io/projected/4010ffa9-1ddd-4f51-afc9-eb5ea50bf969-kube-api-access-hr5ph\") pod \"payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j\" (UID: \"4010ffa9-1ddd-4f51-afc9-eb5ea50bf969\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"
Oct 29 22:34:42.632912 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.617730    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f4g7\" (UniqueName: \"kubernetes.io/projected/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-kube-api-access-9f4g7\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:42.632912 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.617747    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:42.632912 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.617761    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:42.735834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.682390    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4010ffa9-1ddd-4f51-afc9-eb5ea50bf969-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j\" (UID: \"4010ffa9-1ddd-4f51-afc9-eb5ea50bf969\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"
Oct 29 22:34:42.762357 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.762307    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7xq\" (UniqueName: \"kubernetes.io/projected/cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc-kube-api-access-qp7xq\") pod \"payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q\" (UID: \"cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"
Oct 29 22:34:42.762898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.762875    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q\" (UID: \"cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"
Oct 29 22:34:42.771020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.770989    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr5ph\" (UniqueName: \"kubernetes.io/projected/4010ffa9-1ddd-4f51-afc9-eb5ea50bf969-kube-api-access-hr5ph\") pod \"payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j\" (UID: \"4010ffa9-1ddd-4f51-afc9-eb5ea50bf969\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"
Oct 29 22:34:42.906416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.886080    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q\" (UID: \"cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"
Oct 29 22:34:42.906416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.886177    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7xq\" (UniqueName: \"kubernetes.io/projected/cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc-kube-api-access-qp7xq\") pod \"payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q\" (UID: \"cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"
Oct 29 22:34:42.906416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.888780    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q\" (UID: \"cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"
Oct 29 22:34:42.952898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.952676    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"]
Oct 29 22:34:42.960063 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.960026    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e385da9b-7887c9c954-fzglf"]
Oct 29 22:34:42.962235 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.962199    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7xq\" (UniqueName: \"kubernetes.io/projected/cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc-kube-api-access-qp7xq\") pod \"payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q\" (UID: \"cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"
Oct 29 22:34:42.966599 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.966571    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"]
Oct 29 22:34:42.967112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:42.967093    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"
Oct 29 22:34:43.098895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:43.096168    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"
Oct 29 22:34:43.354715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:43.354035    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e8bf7a-abd1-48ee-aa36-462a9e9385bf" path="/var/lib/kubelet/pods/d3e8bf7a-abd1-48ee-aa36-462a9e9385bf/volumes"
Oct 29 22:34:43.567262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:43.567227    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"]
Oct 29 22:34:43.568670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:43.568642    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv" event={"ID":"0400ea45-bb33-492a-9be4-870fe00e0dcf","Type":"ContainerStarted","Data":"c57e60c7ecfc781d67548fcabbe88ffd8442371cc8b0eb6cd22595dd3ae25b37"}
Oct 29 22:34:43.568906 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:43.568880    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv" podUID="0400ea45-bb33-492a-9be4-870fe00e0dcf" containerName="istio-proxy" containerID="cri-o://c57e60c7ecfc781d67548fcabbe88ffd8442371cc8b0eb6cd22595dd3ae25b37" gracePeriod=30
Oct 29 22:34:43.695682 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:34:43.695647    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:34:43.748679 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:43.748627    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"]
Oct 29 22:34:43.752951 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:34:43.752902    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9efeeb_b2b3_48a6_8e7f_e10b493a19fc.slice/crio-49decfb3344c33d915ec5de439b0d9d4cb0429481ff896ebe0a4b820199bddb9 WatchSource:0}: Error finding container 49decfb3344c33d915ec5de439b0d9d4cb0429481ff896ebe0a4b820199bddb9: Status 404 returned error can't find the container with id 49decfb3344c33d915ec5de439b0d9d4cb0429481ff896ebe0a4b820199bddb9
Oct 29 22:34:44.582059 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.582022    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q" event={"ID":"cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc","Type":"ContainerStarted","Data":"a7a1c71a1f5186ca4c4ee25354d88448587bd7a460dfcc26aa6524a8d1e2e857"}
Oct 29 22:34:44.582567 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.582077    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q" event={"ID":"cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc","Type":"ContainerStarted","Data":"49decfb3344c33d915ec5de439b0d9d4cb0429481ff896ebe0a4b820199bddb9"}
Oct 29 22:34:44.582567 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.582237    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"
Oct 29 22:34:44.584007 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.583970    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j" event={"ID":"4010ffa9-1ddd-4f51-afc9-eb5ea50bf969","Type":"ContainerStarted","Data":"8417e84617af0c3654b8c1201b371133789fb1ca6dc17adfec48b18351609a1d"}
Oct 29 22:34:44.584142 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.584011    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j" event={"ID":"4010ffa9-1ddd-4f51-afc9-eb5ea50bf969","Type":"ContainerStarted","Data":"04588957bd1343c959622841b544c9bd9597dbebd7d54f3011aff4438620dbda"}
Oct 29 22:34:44.584142 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.584066    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"
Oct 29 22:34:44.631752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.631697    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q" podStartSLOduration=2.631682617 podStartE2EDuration="2.631682617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:44.627617426 +0000 UTC m=+1223.539060977" watchObservedRunningTime="2026-10-29 22:34:44.631682617 +0000 UTC m=+1223.543126171"
Oct 29 22:34:44.672178 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.672118    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j" podStartSLOduration=2.672101266 podStartE2EDuration="2.672101266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:44.671297641 +0000 UTC m=+1223.582741233" watchObservedRunningTime="2026-10-29 22:34:44.672101266 +0000 UTC m=+1223.583544824"
Oct 29 22:34:44.686918 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.686884    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"]
Oct 29 22:34:44.691380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.691353    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.718168 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.718143    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-aaa3c6-openshift-default-dockercfg-rcl6s\""
Oct 29 22:34:44.728616 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.728046    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"]
Oct 29 22:34:44.843900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.835934    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f479373b-2190-46af-9b6d-ad901981a56b-istiod-ca-cert\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.843900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.835981    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj4bs\" (UniqueName: \"kubernetes.io/projected/f479373b-2190-46af-9b6d-ad901981a56b-kube-api-access-hj4bs\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.843900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.836101    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-credential-socket\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.843900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.836242    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f479373b-2190-46af-9b6d-ad901981a56b-istio-token\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.843900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.836470    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-istio-data\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.843900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.836523    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-workload-certs\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.843900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.836574    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-workload-socket\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.843900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.836608    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f479373b-2190-46af-9b6d-ad901981a56b-istio-podinfo\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.843900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.836642    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-istio-envoy\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.952915 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.952802    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-workload-socket\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.952915 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.952887    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f479373b-2190-46af-9b6d-ad901981a56b-istio-podinfo\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.952915 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.952919    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-istio-envoy\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.953549 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.953056    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f479373b-2190-46af-9b6d-ad901981a56b-istiod-ca-cert\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.953549 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.953086    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj4bs\" (UniqueName: \"kubernetes.io/projected/f479373b-2190-46af-9b6d-ad901981a56b-kube-api-access-hj4bs\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.953549 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.953212    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-credential-socket\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.953549 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.953355    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f479373b-2190-46af-9b6d-ad901981a56b-istio-token\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.953549 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.953410    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-istio-data\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.953549 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.953447    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-workload-certs\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.958838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.953936    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-workload-certs\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.958838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.954202    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-workload-socket\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.961844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.961014    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-credential-socket\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.961844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.961284    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-istio-data\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:44.993330 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:44.993293    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f479373b-2190-46af-9b6d-ad901981a56b-istiod-ca-cert\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:45.001284 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:45.001208    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f479373b-2190-46af-9b6d-ad901981a56b-istio-token\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:45.004389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:45.004320    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj4bs\" (UniqueName: \"kubernetes.io/projected/f479373b-2190-46af-9b6d-ad901981a56b-kube-api-access-hj4bs\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:45.009762 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:45.007860    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f479373b-2190-46af-9b6d-ad901981a56b-istio-podinfo\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:45.045235 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:45.045141    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-istio-envoy\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:45.312764 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:45.312725    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:45.660680 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:34:45.660649    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf479373b_2190_46af_9b6d_ad901981a56b.slice/crio-ffe798ab6b4b31a95f4ca07156ebd00c723bb15406c92c2e16b47dbb05738ceb WatchSource:0}: Error finding container ffe798ab6b4b31a95f4ca07156ebd00c723bb15406c92c2e16b47dbb05738ceb: Status 404 returned error can't find the container with id ffe798ab6b4b31a95f4ca07156ebd00c723bb15406c92c2e16b47dbb05738ceb
Oct 29 22:34:45.661140 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:45.660854    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"]
Oct 29 22:34:45.663550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:45.663517    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:34:45.663665 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:45.663582    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:34:45.663665 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:45.663613    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:34:46.681402 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:46.681357    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m" event={"ID":"f479373b-2190-46af-9b6d-ad901981a56b","Type":"ContainerStarted","Data":"24dfc707efe15a6ea6c7fb90c9a6b6461ceccf29b006803f913d700c006b1cf7"}
Oct 29 22:34:46.683499 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:46.683464    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m" event={"ID":"f479373b-2190-46af-9b6d-ad901981a56b","Type":"ContainerStarted","Data":"ffe798ab6b4b31a95f4ca07156ebd00c723bb15406c92c2e16b47dbb05738ceb"}
Oct 29 22:34:46.745606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:46.745550    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m" podStartSLOduration=2.7455352619999998 podStartE2EDuration="2.745535262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:46.742252053 +0000 UTC m=+1225.653695607" watchObservedRunningTime="2026-10-29 22:34:46.745535262 +0000 UTC m=+1225.656978816"
Oct 29 22:34:47.313623 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:47.313593    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:47.319122 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:47.319094    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:47.688990 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:47.688951    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:47.690013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:47.689990    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:34:48.698425 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.698384    4559 generic.go:356] "Generic (PLEG): container finished" podID="0400ea45-bb33-492a-9be4-870fe00e0dcf" containerID="c57e60c7ecfc781d67548fcabbe88ffd8442371cc8b0eb6cd22595dd3ae25b37" exitCode=0
Oct 29 22:34:48.698941 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.698425    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv" event={"ID":"0400ea45-bb33-492a-9be4-870fe00e0dcf","Type":"ContainerDied","Data":"c57e60c7ecfc781d67548fcabbe88ffd8442371cc8b0eb6cd22595dd3ae25b37"}
Oct 29 22:34:48.698941 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.698516    4559 scope.go:122] "RemoveContainer" containerID="1734268123b155968b7ad947c04035cd416b1f06e04554f64d4ec2f2af1bc9b5"
Oct 29 22:34:48.877195 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.877167    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:34:48.976617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.976571    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0400ea45-bb33-492a-9be4-870fe00e0dcf-kube-api-access-tjkz7\" (UniqueName: \"kubernetes.io/projected/0400ea45-bb33-492a-9be4-870fe00e0dcf-kube-api-access-tjkz7\") pod \"0400ea45-bb33-492a-9be4-870fe00e0dcf\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") "
Oct 29 22:34:48.976866 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.976754    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-credential-socket\") pod \"0400ea45-bb33-492a-9be4-870fe00e0dcf\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") "
Oct 29 22:34:48.976866 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.976838    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-data\") pod \"0400ea45-bb33-492a-9be4-870fe00e0dcf\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") "
Oct 29 22:34:48.977024 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.977005    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0400ea45-bb33-492a-9be4-870fe00e0dcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0400ea45-bb33-492a-9be4-870fe00e0dcf-istiod-ca-cert\") pod \"0400ea45-bb33-492a-9be4-870fe00e0dcf\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") "
Oct 29 22:34:48.977082 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.977056    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-workload-socket\") pod \"0400ea45-bb33-492a-9be4-870fe00e0dcf\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") "
Oct 29 22:34:48.977137 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.977086    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-token\" (UniqueName: \"kubernetes.io/projected/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-token\") pod \"0400ea45-bb33-492a-9be4-870fe00e0dcf\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") "
Oct 29 22:34:48.977194 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.977133    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-workload-certs\") pod \"0400ea45-bb33-492a-9be4-870fe00e0dcf\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") "
Oct 29 22:34:48.977194 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.977176    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-envoy\") pod \"0400ea45-bb33-492a-9be4-870fe00e0dcf\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") "
Oct 29 22:34:48.977312 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.977295    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-podinfo\") pod \"0400ea45-bb33-492a-9be4-870fe00e0dcf\" (UID: \"0400ea45-bb33-492a-9be4-870fe00e0dcf\") "
Oct 29 22:34:48.977458 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.977428    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0400ea45-bb33-492a-9be4-870fe00e0dcf-istiod-ca-cert" pod "0400ea45-bb33-492a-9be4-870fe00e0dcf" (UID: "0400ea45-bb33-492a-9be4-870fe00e0dcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:34:48.977741 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.977711    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-workload-certs" pod "0400ea45-bb33-492a-9be4-870fe00e0dcf" (UID: "0400ea45-bb33-492a-9be4-870fe00e0dcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:34:48.977741 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.977733    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-workload-socket" pod "0400ea45-bb33-492a-9be4-870fe00e0dcf" (UID: "0400ea45-bb33-492a-9be4-870fe00e0dcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:34:48.978422 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.978128    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0400ea45-bb33-492a-9be4-870fe00e0dcf-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:48.978422 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.978158    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:48.978422 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.978172    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:48.978422 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.978388    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-credential-socket" pod "0400ea45-bb33-492a-9be4-870fe00e0dcf" (UID: "0400ea45-bb33-492a-9be4-870fe00e0dcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:34:48.978866 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.978761    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-data" pod "0400ea45-bb33-492a-9be4-870fe00e0dcf" (UID: "0400ea45-bb33-492a-9be4-870fe00e0dcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:34:48.980520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.980491    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0400ea45-bb33-492a-9be4-870fe00e0dcf-kube-api-access-tjkz7" pod "0400ea45-bb33-492a-9be4-870fe00e0dcf" (UID: "0400ea45-bb33-492a-9be4-870fe00e0dcf"). InnerVolumeSpecName "kube-api-access-tjkz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:34:48.980702 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.980679    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-podinfo" pod "0400ea45-bb33-492a-9be4-870fe00e0dcf" (UID: "0400ea45-bb33-492a-9be4-870fe00e0dcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:34:48.981430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.981399    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-token" pod "0400ea45-bb33-492a-9be4-870fe00e0dcf" (UID: "0400ea45-bb33-492a-9be4-870fe00e0dcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:34:48.981525 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:48.981474    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-envoy" pod "0400ea45-bb33-492a-9be4-870fe00e0dcf" (UID: "0400ea45-bb33-492a-9be4-870fe00e0dcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:34:49.080046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:49.080001    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:49.080046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:49.080038    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:49.080046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:49.080052    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:49.080274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:49.080065    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjkz7\" (UniqueName: \"kubernetes.io/projected/0400ea45-bb33-492a-9be4-870fe00e0dcf-kube-api-access-tjkz7\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:49.080274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:49.080090    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:49.080274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:49.080103    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0400ea45-bb33-492a-9be4-870fe00e0dcf-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:34:49.706899 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:49.706863    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"
Oct 29 22:34:49.707306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:49.706862    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv" event={"ID":"0400ea45-bb33-492a-9be4-870fe00e0dcf","Type":"ContainerDied","Data":"1aafa1bed2367951855b13dff576e0f88121cd61ee696cfed9d5a39ebb5df434"}
Oct 29 22:34:49.707306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:49.706998    4559 scope.go:122] "RemoveContainer" containerID="c57e60c7ecfc781d67548fcabbe88ffd8442371cc8b0eb6cd22595dd3ae25b37"
Oct 29 22:34:49.762445 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:49.762413    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"]
Oct 29 22:34:49.777384 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:49.777345    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e385da9b-gw-openshift-default-6d84d6474f-mljgv"]
Oct 29 22:34:51.013684 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.013648    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"]
Oct 29 22:34:51.014338 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.014320    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0400ea45-bb33-492a-9be4-870fe00e0dcf" containerName="istio-proxy"
Oct 29 22:34:51.014338 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.014337    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0400ea45-bb33-492a-9be4-870fe00e0dcf" containerName="istio-proxy"
Oct 29 22:34:51.014506 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.014466    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0400ea45-bb33-492a-9be4-870fe00e0dcf" containerName="istio-proxy"
Oct 29 22:34:51.014506 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.014476    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0400ea45-bb33-492a-9be4-870fe00e0dcf" containerName="istio-proxy"
Oct 29 22:34:51.022920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.022893    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"
Oct 29 22:34:51.039436 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.039404    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"]
Oct 29 22:34:51.046171 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.046137    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-aaa3c6-dockercfg-9pr9n\""
Oct 29 22:34:51.086124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.086086    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-aaa3c6\""
Oct 29 22:34:51.141838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.138827    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjdhf\" (UniqueName: \"kubernetes.io/projected/d838fae2-87e9-4874-bb79-92a34e503ce6-kube-api-access-sjdhf\") pod \"payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m\" (UID: \"d838fae2-87e9-4874-bb79-92a34e503ce6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"
Oct 29 22:34:51.141838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.138974    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d838fae2-87e9-4874-bb79-92a34e503ce6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m\" (UID: \"d838fae2-87e9-4874-bb79-92a34e503ce6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"
Oct 29 22:34:51.258836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.241028    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjdhf\" (UniqueName: \"kubernetes.io/projected/d838fae2-87e9-4874-bb79-92a34e503ce6-kube-api-access-sjdhf\") pod \"payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m\" (UID: \"d838fae2-87e9-4874-bb79-92a34e503ce6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"
Oct 29 22:34:51.283890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.259242    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d838fae2-87e9-4874-bb79-92a34e503ce6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m\" (UID: \"d838fae2-87e9-4874-bb79-92a34e503ce6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"
Oct 29 22:34:51.480825 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.480745    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d838fae2-87e9-4874-bb79-92a34e503ce6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m\" (UID: \"d838fae2-87e9-4874-bb79-92a34e503ce6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"
Oct 29 22:34:51.486551 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.486463    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjdhf\" (UniqueName: \"kubernetes.io/projected/d838fae2-87e9-4874-bb79-92a34e503ce6-kube-api-access-sjdhf\") pod \"payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m\" (UID: \"d838fae2-87e9-4874-bb79-92a34e503ce6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"
Oct 29 22:34:51.496848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.496790    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0400ea45-bb33-492a-9be4-870fe00e0dcf" path="/var/lib/kubelet/pods/0400ea45-bb33-492a-9be4-870fe00e0dcf/volumes"
Oct 29 22:34:51.652656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.652616    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"
Oct 29 22:34:51.968171 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:34:51.968086    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd838fae2_87e9_4874_bb79_92a34e503ce6.slice/crio-c47799a4267e0849df316dded57508355f17b439c6e036dbd832173bcc770624 WatchSource:0}: Error finding container c47799a4267e0849df316dded57508355f17b439c6e036dbd832173bcc770624: Status 404 returned error can't find the container with id c47799a4267e0849df316dded57508355f17b439c6e036dbd832173bcc770624
Oct 29 22:34:51.970172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:51.970131    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"]
Oct 29 22:34:52.737582 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:52.737537    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m" event={"ID":"d838fae2-87e9-4874-bb79-92a34e503ce6","Type":"ContainerStarted","Data":"f302bc68b25b097a604a22004fd9d01e628990029018c4e357687848d577b27c"}
Oct 29 22:34:52.738119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:52.737605    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m" event={"ID":"d838fae2-87e9-4874-bb79-92a34e503ce6","Type":"ContainerStarted","Data":"c47799a4267e0849df316dded57508355f17b439c6e036dbd832173bcc770624"}
Oct 29 22:34:52.738119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:52.737641    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"
Oct 29 22:34:52.781558 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:52.781487    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m" podStartSLOduration=2.781465693 podStartE2EDuration="2.781465693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:52.777432795 +0000 UTC m=+1231.688876350" watchObservedRunningTime="2026-10-29 22:34:52.781465693 +0000 UTC m=+1231.692909247"
Oct 29 22:34:55.246004 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.245965    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"]
Oct 29 22:34:55.246999 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.246977    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0400ea45-bb33-492a-9be4-870fe00e0dcf" containerName="istio-proxy"
Oct 29 22:34:55.247045 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.247002    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0400ea45-bb33-492a-9be4-870fe00e0dcf" containerName="istio-proxy"
Oct 29 22:34:55.251690 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.251665    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:34:55.272410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.272377    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"]
Oct 29 22:34:55.278893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.278837    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-06a95f-serving-cert\""
Oct 29 22:34:55.360107 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.360072    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a75464c2-c1ca-4991-8634-7c1f417400b5-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl\" (UID: \"a75464c2-c1ca-4991-8634-7c1f417400b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:34:55.360299 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.360128    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75464c2-c1ca-4991-8634-7c1f417400b5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl\" (UID: \"a75464c2-c1ca-4991-8634-7c1f417400b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:34:55.360299 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.360248    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c87b6\" (UniqueName: \"kubernetes.io/projected/a75464c2-c1ca-4991-8634-7c1f417400b5-kube-api-access-c87b6\") pod \"maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl\" (UID: \"a75464c2-c1ca-4991-8634-7c1f417400b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:34:55.360372 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.360305    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75464c2-c1ca-4991-8634-7c1f417400b5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl\" (UID: \"a75464c2-c1ca-4991-8634-7c1f417400b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:34:55.462275 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.462225    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87b6\" (UniqueName: \"kubernetes.io/projected/a75464c2-c1ca-4991-8634-7c1f417400b5-kube-api-access-c87b6\") pod \"maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl\" (UID: \"a75464c2-c1ca-4991-8634-7c1f417400b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:34:55.462505 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.462339    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75464c2-c1ca-4991-8634-7c1f417400b5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl\" (UID: \"a75464c2-c1ca-4991-8634-7c1f417400b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:34:55.462624 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.462607    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a75464c2-c1ca-4991-8634-7c1f417400b5-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl\" (UID: \"a75464c2-c1ca-4991-8634-7c1f417400b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:34:55.462686 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.462656    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75464c2-c1ca-4991-8634-7c1f417400b5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl\" (UID: \"a75464c2-c1ca-4991-8634-7c1f417400b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:34:55.463468 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.463435    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75464c2-c1ca-4991-8634-7c1f417400b5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl\" (UID: \"a75464c2-c1ca-4991-8634-7c1f417400b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:34:55.465182 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.465160    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a75464c2-c1ca-4991-8634-7c1f417400b5-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl\" (UID: \"a75464c2-c1ca-4991-8634-7c1f417400b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:34:55.465352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.465330    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75464c2-c1ca-4991-8634-7c1f417400b5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl\" (UID: \"a75464c2-c1ca-4991-8634-7c1f417400b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:34:55.482652 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.482612    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87b6\" (UniqueName: \"kubernetes.io/projected/a75464c2-c1ca-4991-8634-7c1f417400b5-kube-api-access-c87b6\") pod \"maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl\" (UID: \"a75464c2-c1ca-4991-8634-7c1f417400b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:34:55.568301 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.568202    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:34:55.597270 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.597241    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"
Oct 29 22:34:55.599162 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:55.599139    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"
Oct 29 22:34:55.998601 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:34:55.998569    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75464c2_c1ca_4991_8634_7c1f417400b5.slice/crio-1cd7ee959bc9340750566d3358d3895035c12de40f116584af98be73086e68ff WatchSource:0}: Error finding container 1cd7ee959bc9340750566d3358d3895035c12de40f116584af98be73086e68ff: Status 404 returned error can't find the container with id 1cd7ee959bc9340750566d3358d3895035c12de40f116584af98be73086e68ff
Oct 29 22:34:56.000605 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:56.000573    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"]
Oct 29 22:34:56.797599 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:56.797487    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl" event={"ID":"a75464c2-c1ca-4991-8634-7c1f417400b5","Type":"ContainerStarted","Data":"a40cf86174a72fa8d8a2faf133a965e12984d0ccfb73cc8838052f1fbe086487"}
Oct 29 22:34:56.797599 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:56.797569    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:34:56.797599 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:56.797595    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl" event={"ID":"a75464c2-c1ca-4991-8634-7c1f417400b5","Type":"ContainerStarted","Data":"1cd7ee959bc9340750566d3358d3895035c12de40f116584af98be73086e68ff"}
Oct 29 22:34:56.851880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:56.839755    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl" podStartSLOduration=1.368793647 podStartE2EDuration="1.839735749s" totalImagesPullingTime="470.942102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:56.837035414 +0000 UTC m=+1235.748478979" watchObservedRunningTime="2026-10-29 22:34:56.839735749 +0000 UTC m=+1235.751179307"
Oct 29 22:34:58.806716 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:58.806676    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"]
Oct 29 22:34:58.838495 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:34:58.838462    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"]
Oct 29 22:35:02.295616 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:02.295519    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"]
Oct 29 22:35:02.297405 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:02.295801    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5" podUID="158aa4ff-b698-4c6e-b6c5-a3399296dad5" containerName="maas-api" containerID="cri-o://c7777adbb235f3f7deea554e223dcf48aa85d60d74920099b2d5037ab7e1513d" gracePeriod=30
Oct 29 22:35:02.395450 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:02.385082    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5" podUID="158aa4ff-b698-4c6e-b6c5-a3399296dad5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.90:8443/health\": dial tcp 10.134.0.90:8443: connect: connection refused"
Oct 29 22:35:02.860313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:02.851209    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:35:02.945980 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:02.941728    4559 generic.go:356] "Generic (PLEG): container finished" podID="158aa4ff-b698-4c6e-b6c5-a3399296dad5" containerID="c7777adbb235f3f7deea554e223dcf48aa85d60d74920099b2d5037ab7e1513d" exitCode=0
Oct 29 22:35:02.945980 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:02.941968    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5" event={"ID":"158aa4ff-b698-4c6e-b6c5-a3399296dad5","Type":"ContainerDied","Data":"c7777adbb235f3f7deea554e223dcf48aa85d60d74920099b2d5037ab7e1513d"}
Oct 29 22:35:03.335897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.331104    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:35:03.620608 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.620478    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/158aa4ff-b698-4c6e-b6c5-a3399296dad5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/158aa4ff-b698-4c6e-b6c5-a3399296dad5-maas-api-tls\") pod \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\" (UID: \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\") "
Oct 29 22:35:03.639179 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.639146    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158aa4ff-b698-4c6e-b6c5-a3399296dad5-kube-api-access-pq8rv\" (UniqueName: \"kubernetes.io/projected/158aa4ff-b698-4c6e-b6c5-a3399296dad5-kube-api-access-pq8rv\") pod \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\" (UID: \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\") "
Oct 29 22:35:03.641066 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.641007    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/158aa4ff-b698-4c6e-b6c5-a3399296dad5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/158aa4ff-b698-4c6e-b6c5-a3399296dad5-metrics-certs\") pod \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\" (UID: \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\") "
Oct 29 22:35:03.643950 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.643902    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158aa4ff-b698-4c6e-b6c5-a3399296dad5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/158aa4ff-b698-4c6e-b6c5-a3399296dad5-openshift-service-ca\") pod \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\" (UID: \"158aa4ff-b698-4c6e-b6c5-a3399296dad5\") "
Oct 29 22:35:03.644572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.644539    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158aa4ff-b698-4c6e-b6c5-a3399296dad5-openshift-service-ca" pod "158aa4ff-b698-4c6e-b6c5-a3399296dad5" (UID: "158aa4ff-b698-4c6e-b6c5-a3399296dad5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:35:03.651328 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.650290    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/158aa4ff-b698-4c6e-b6c5-a3399296dad5-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:35:03.669884 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.669766    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/158aa4ff-b698-4c6e-b6c5-a3399296dad5-maas-api-tls" pod "158aa4ff-b698-4c6e-b6c5-a3399296dad5" (UID: "158aa4ff-b698-4c6e-b6c5-a3399296dad5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:35:03.675940 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.675898    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158aa4ff-b698-4c6e-b6c5-a3399296dad5-kube-api-access-pq8rv" pod "158aa4ff-b698-4c6e-b6c5-a3399296dad5" (UID: "158aa4ff-b698-4c6e-b6c5-a3399296dad5"). InnerVolumeSpecName "kube-api-access-pq8rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:35:03.676489 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.676462    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/158aa4ff-b698-4c6e-b6c5-a3399296dad5-metrics-certs" pod "158aa4ff-b698-4c6e-b6c5-a3399296dad5" (UID: "158aa4ff-b698-4c6e-b6c5-a3399296dad5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:35:03.748986 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.748958    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"
Oct 29 22:35:03.751532 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.751497    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158aa4ff-b698-4c6e-b6c5-a3399296dad5-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:35:03.751647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.751537    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/158aa4ff-b698-4c6e-b6c5-a3399296dad5-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:35:03.751647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.751551    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq8rv\" (UniqueName: \"kubernetes.io/projected/158aa4ff-b698-4c6e-b6c5-a3399296dad5-kube-api-access-pq8rv\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:35:03.807798 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.807748    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh" podUID="72172fd1-33a9-4322-9d65-e923277c3b51" containerName="payload-processing" containerID="cri-o://b8602a982d0ece76f0a1d02c03854735b00c7ea752120db45743e936a908eb7f" gracePeriod=25
Oct 29 22:35:03.839846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.839747    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn" podUID="bcf9d46d-ec5f-407c-bb46-563f65581bed" containerName="payload-pre-processing" containerID="cri-o://2057b14fc7532a9f8825999e44fb4baac8dcd5d8987a117702942bd907c0a37b" gracePeriod=25
Oct 29 22:35:03.958792 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.958623    4559 generic.go:356] "Generic (PLEG): container finished" podID="72172fd1-33a9-4322-9d65-e923277c3b51" containerID="b8602a982d0ece76f0a1d02c03854735b00c7ea752120db45743e936a908eb7f" exitCode=0
Oct 29 22:35:03.959304 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.959275    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh" event={"ID":"72172fd1-33a9-4322-9d65-e923277c3b51","Type":"ContainerDied","Data":"b8602a982d0ece76f0a1d02c03854735b00c7ea752120db45743e936a908eb7f"}
Oct 29 22:35:03.969756 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.969719    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5" event={"ID":"158aa4ff-b698-4c6e-b6c5-a3399296dad5","Type":"ContainerDied","Data":"f0ad0aa8c791f2278bb204249bdfb7c67b420c249427d2414ea7f36916ec2c42"}
Oct 29 22:35:03.969933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.969793    4559 scope.go:122] "RemoveContainer" containerID="c7777adbb235f3f7deea554e223dcf48aa85d60d74920099b2d5037ab7e1513d"
Oct 29 22:35:03.970024 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:03.970006    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"
Oct 29 22:35:04.041015 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.040241    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"]
Oct 29 22:35:04.046496 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.046205    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c6e65e5-6444467978-7tst5"]
Oct 29 22:35:04.104572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.104546    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"
Oct 29 22:35:04.185931 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.169311    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72172fd1-33a9-4322-9d65-e923277c3b51-kube-api-access-lkldp\" (UniqueName: \"kubernetes.io/projected/72172fd1-33a9-4322-9d65-e923277c3b51-kube-api-access-lkldp\") pod \"72172fd1-33a9-4322-9d65-e923277c3b51\" (UID: \"72172fd1-33a9-4322-9d65-e923277c3b51\") "
Oct 29 22:35:04.185931 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.169484    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72172fd1-33a9-4322-9d65-e923277c3b51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72172fd1-33a9-4322-9d65-e923277c3b51-plugins-config-volume\") pod \"72172fd1-33a9-4322-9d65-e923277c3b51\" (UID: \"72172fd1-33a9-4322-9d65-e923277c3b51\") "
Oct 29 22:35:04.189208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.186482    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72172fd1-33a9-4322-9d65-e923277c3b51-plugins-config-volume" pod "72172fd1-33a9-4322-9d65-e923277c3b51" (UID: "72172fd1-33a9-4322-9d65-e923277c3b51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:35:04.193649 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.193592    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72172fd1-33a9-4322-9d65-e923277c3b51-kube-api-access-lkldp" pod "72172fd1-33a9-4322-9d65-e923277c3b51" (UID: "72172fd1-33a9-4322-9d65-e923277c3b51"). InnerVolumeSpecName "kube-api-access-lkldp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:35:04.203005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.202971    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"
Oct 29 22:35:04.271744 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.271705    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf9d46d-ec5f-407c-bb46-563f65581bed-kube-api-access-xz66w\" (UniqueName: \"kubernetes.io/projected/bcf9d46d-ec5f-407c-bb46-563f65581bed-kube-api-access-xz66w\") pod \"bcf9d46d-ec5f-407c-bb46-563f65581bed\" (UID: \"bcf9d46d-ec5f-407c-bb46-563f65581bed\") "
Oct 29 22:35:04.271984 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.271790    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf9d46d-ec5f-407c-bb46-563f65581bed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcf9d46d-ec5f-407c-bb46-563f65581bed-plugins-config-volume\") pod \"bcf9d46d-ec5f-407c-bb46-563f65581bed\" (UID: \"bcf9d46d-ec5f-407c-bb46-563f65581bed\") "
Oct 29 22:35:04.272267 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.272239    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf9d46d-ec5f-407c-bb46-563f65581bed-plugins-config-volume" pod "bcf9d46d-ec5f-407c-bb46-563f65581bed" (UID: "bcf9d46d-ec5f-407c-bb46-563f65581bed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:35:04.272848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.272800    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72172fd1-33a9-4322-9d65-e923277c3b51-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:35:04.272848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.272850    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcf9d46d-ec5f-407c-bb46-563f65581bed-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:35:04.273034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.272864    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkldp\" (UniqueName: \"kubernetes.io/projected/72172fd1-33a9-4322-9d65-e923277c3b51-kube-api-access-lkldp\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:35:04.275147 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.275112    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf9d46d-ec5f-407c-bb46-563f65581bed-kube-api-access-xz66w" pod "bcf9d46d-ec5f-407c-bb46-563f65581bed" (UID: "bcf9d46d-ec5f-407c-bb46-563f65581bed"). InnerVolumeSpecName "kube-api-access-xz66w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:35:04.374061 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.374024    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz66w\" (UniqueName: \"kubernetes.io/projected/bcf9d46d-ec5f-407c-bb46-563f65581bed-kube-api-access-xz66w\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:35:04.705262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.705230    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"]
Oct 29 22:35:04.705917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.705903    4559 cpu_manager.go:414] "containerMap: removing container" podUID="158aa4ff-b698-4c6e-b6c5-a3399296dad5" containerName="maas-api"
Oct 29 22:35:04.705976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.705918    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158aa4ff-b698-4c6e-b6c5-a3399296dad5" containerName="maas-api"
Oct 29 22:35:04.705976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.705937    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bcf9d46d-ec5f-407c-bb46-563f65581bed" containerName="payload-pre-processing"
Oct 29 22:35:04.705976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.705941    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf9d46d-ec5f-407c-bb46-563f65581bed" containerName="payload-pre-processing"
Oct 29 22:35:04.705976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.705960    4559 cpu_manager.go:414] "containerMap: removing container" podUID="72172fd1-33a9-4322-9d65-e923277c3b51" containerName="payload-processing"
Oct 29 22:35:04.705976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.705964    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72172fd1-33a9-4322-9d65-e923277c3b51" containerName="payload-processing"
Oct 29 22:35:04.706126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.706045    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="72172fd1-33a9-4322-9d65-e923277c3b51" containerName="payload-processing"
Oct 29 22:35:04.706126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.706060    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="158aa4ff-b698-4c6e-b6c5-a3399296dad5" containerName="maas-api"
Oct 29 22:35:04.706126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.706067    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf9d46d-ec5f-407c-bb46-563f65581bed" containerName="payload-pre-processing"
Oct 29 22:35:04.709787 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.709767    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 22:35:04.732862 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.732826    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"]
Oct 29 22:35:04.747860 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.743736    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-aaa3c6-serving-cert\""
Oct 29 22:35:04.780976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.780937    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e824b452-5c7a-4d42-8f86-bf43dd571680-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n\" (UID: \"e824b452-5c7a-4d42-8f86-bf43dd571680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 22:35:04.781151 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.781007    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e824b452-5c7a-4d42-8f86-bf43dd571680-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n\" (UID: \"e824b452-5c7a-4d42-8f86-bf43dd571680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 22:35:04.781263 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.781244    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e824b452-5c7a-4d42-8f86-bf43dd571680-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n\" (UID: \"e824b452-5c7a-4d42-8f86-bf43dd571680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 22:35:04.781630 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.781604    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp8gb\" (UniqueName: \"kubernetes.io/projected/e824b452-5c7a-4d42-8f86-bf43dd571680-kube-api-access-dp8gb\") pod \"maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n\" (UID: \"e824b452-5c7a-4d42-8f86-bf43dd571680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 22:35:04.883232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.883193    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e824b452-5c7a-4d42-8f86-bf43dd571680-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n\" (UID: \"e824b452-5c7a-4d42-8f86-bf43dd571680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 22:35:04.883402 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.883350    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp8gb\" (UniqueName: \"kubernetes.io/projected/e824b452-5c7a-4d42-8f86-bf43dd571680-kube-api-access-dp8gb\") pod \"maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n\" (UID: \"e824b452-5c7a-4d42-8f86-bf43dd571680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 22:35:04.883463 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.883399    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e824b452-5c7a-4d42-8f86-bf43dd571680-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n\" (UID: \"e824b452-5c7a-4d42-8f86-bf43dd571680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 22:35:04.883463 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.883444    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e824b452-5c7a-4d42-8f86-bf43dd571680-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n\" (UID: \"e824b452-5c7a-4d42-8f86-bf43dd571680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 22:35:04.884345 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.884321    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e824b452-5c7a-4d42-8f86-bf43dd571680-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n\" (UID: \"e824b452-5c7a-4d42-8f86-bf43dd571680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 22:35:04.885961 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.885941    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e824b452-5c7a-4d42-8f86-bf43dd571680-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n\" (UID: \"e824b452-5c7a-4d42-8f86-bf43dd571680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 22:35:04.886153 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.886133    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e824b452-5c7a-4d42-8f86-bf43dd571680-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n\" (UID: \"e824b452-5c7a-4d42-8f86-bf43dd571680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 22:35:04.904540 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:04.904504    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp8gb\" (UniqueName: \"kubernetes.io/projected/e824b452-5c7a-4d42-8f86-bf43dd571680-kube-api-access-dp8gb\") pod \"maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n\" (UID: \"e824b452-5c7a-4d42-8f86-bf43dd571680\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 22:35:05.032895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:05.025646    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 22:35:05.114655 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:05.108474    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh" event={"ID":"72172fd1-33a9-4322-9d65-e923277c3b51","Type":"ContainerDied","Data":"5ce6100fcc69ac31f811a92ff43984d8840f9e14a12c4c7722d833a8ff83685a"}
Oct 29 22:35:05.114655 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:05.108574    4559 scope.go:122] "RemoveContainer" containerID="b8602a982d0ece76f0a1d02c03854735b00c7ea752120db45743e936a908eb7f"
Oct 29 22:35:05.114655 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:05.109078    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"
Oct 29 22:35:05.139027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:05.130203    4559 generic.go:356] "Generic (PLEG): container finished" podID="bcf9d46d-ec5f-407c-bb46-563f65581bed" containerID="2057b14fc7532a9f8825999e44fb4baac8dcd5d8987a117702942bd907c0a37b" exitCode=0
Oct 29 22:35:05.139027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:05.130422    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"
Oct 29 22:35:05.139027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:05.131232    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn" event={"ID":"bcf9d46d-ec5f-407c-bb46-563f65581bed","Type":"ContainerDied","Data":"2057b14fc7532a9f8825999e44fb4baac8dcd5d8987a117702942bd907c0a37b"}
Oct 29 22:35:05.139027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:05.131295    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn" event={"ID":"bcf9d46d-ec5f-407c-bb46-563f65581bed","Type":"ContainerDied","Data":"a746c810001a55d8a1ccca7201d5d1d21ae002482bdc14902b02345c0aa966f4"}
Oct 29 22:35:05.185836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:05.179091    4559 scope.go:122] "RemoveContainer" containerID="2057b14fc7532a9f8825999e44fb4baac8dcd5d8987a117702942bd907c0a37b"
Oct 29 22:35:05.203963 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:05.203925    4559 scope.go:122] "RemoveContainer" containerID="2057b14fc7532a9f8825999e44fb4baac8dcd5d8987a117702942bd907c0a37b"
Oct 29 22:35:05.204477 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:35:05.204373    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2057b14fc7532a9f8825999e44fb4baac8dcd5d8987a117702942bd907c0a37b\": container with ID starting with 2057b14fc7532a9f8825999e44fb4baac8dcd5d8987a117702942bd907c0a37b not found: ID does not exist" containerID="2057b14fc7532a9f8825999e44fb4baac8dcd5d8987a117702942bd907c0a37b"
Oct 29 22:35:05.204549 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:05.204493    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2057b14fc7532a9f8825999e44fb4baac8dcd5d8987a117702942bd907c0a37b"} err="failed to get container status \"2057b14fc7532a9f8825999e44fb4baac8dcd5d8987a117702942bd907c0a37b\": rpc error: code = NotFound desc = could not find container \"2057b14fc7532a9f8825999e44fb4baac8dcd5d8987a117702942bd907c0a37b\": container with ID starting with 2057b14fc7532a9f8825999e44fb4baac8dcd5d8987a117702942bd907c0a37b not found: ID does not exist"
Oct 29 22:35:05.238663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:05.238628    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158aa4ff-b698-4c6e-b6c5-a3399296dad5" path="/var/lib/kubelet/pods/158aa4ff-b698-4c6e-b6c5-a3399296dad5/volumes"
Oct 29 22:35:05.239366 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:05.239327    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"]
Oct 29 22:35:05.239582 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:05.239567    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c6e65e5-6c89f86574-szfbh"]
Oct 29 22:35:05.278186 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:05.278084    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"]
Oct 29 22:35:05.287640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:05.287549    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c6e65e5-78df9fdc95-qrgvn"]
Oct 29 22:35:05.483325 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:35:05.483288    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode824b452_5c7a_4d42_8f86_bf43dd571680.slice/crio-72a9589ede96f21d6045bab3e40b9803e9fb07db210d91bb13ba6a3682ab7e32 WatchSource:0}: Error finding container 72a9589ede96f21d6045bab3e40b9803e9fb07db210d91bb13ba6a3682ab7e32: Status 404 returned error can't find the container with id 72a9589ede96f21d6045bab3e40b9803e9fb07db210d91bb13ba6a3682ab7e32
Oct 29 22:35:05.483789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:05.483765    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"]
Oct 29 22:35:06.297833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:06.292796    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n" event={"ID":"e824b452-5c7a-4d42-8f86-bf43dd571680","Type":"ContainerStarted","Data":"61be77f7d8d7c8b15fbef90d1b8cb248298c02ba5e771db55f79f4449bd6cca8"}
Oct 29 22:35:06.297833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:06.292905    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n" event={"ID":"e824b452-5c7a-4d42-8f86-bf43dd571680","Type":"ContainerStarted","Data":"72a9589ede96f21d6045bab3e40b9803e9fb07db210d91bb13ba6a3682ab7e32"}
Oct 29 22:35:06.297833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:06.294500    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 22:35:06.384417 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:06.375206    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n" podStartSLOduration=2.003261175 podStartE2EDuration="2.375183831s" totalImagesPullingTime="371.922656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:06.368481827 +0000 UTC m=+1245.279925385" watchObservedRunningTime="2026-10-29 22:35:06.375183831 +0000 UTC m=+1245.286627386"
Oct 29 22:35:07.232180 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:07.232135    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72172fd1-33a9-4322-9d65-e923277c3b51" path="/var/lib/kubelet/pods/72172fd1-33a9-4322-9d65-e923277c3b51/volumes"
Oct 29 22:35:07.232847 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:07.232785    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf9d46d-ec5f-407c-bb46-563f65581bed" path="/var/lib/kubelet/pods/bcf9d46d-ec5f-407c-bb46-563f65581bed/volumes"
Oct 29 22:35:08.441539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.441504    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"]
Oct 29 22:35:08.482403 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.482369    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"]
Oct 29 22:35:08.482577 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.482510    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.504110 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.504067    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1131a29d-openshift-default-dockercfg-8jjv9\""
Oct 29 22:35:08.624887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.621127    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-workload-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.624887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.621202    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-podinfo\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.624887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.621282    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istiod-ca-cert\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.624887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.621310    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-data\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.624887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.621374    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-token\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.624887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.621494    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-workload-certs\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.624887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.621565    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-credential-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.624887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.621631    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j42hv\" (UniqueName: \"kubernetes.io/projected/0d6a7797-f46f-45bd-a2ca-47736b909cf9-kube-api-access-j42hv\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.624887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.621677    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-envoy\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.730892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.729739    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j42hv\" (UniqueName: \"kubernetes.io/projected/0d6a7797-f46f-45bd-a2ca-47736b909cf9-kube-api-access-j42hv\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.730892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.729858    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-envoy\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.730892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.730007    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-workload-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.730892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.730050    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-podinfo\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.730892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.730123    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istiod-ca-cert\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.730892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.730152    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-data\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.730892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.730211    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-token\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.730892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.730314    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-workload-certs\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.730892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.730382    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-credential-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.730892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.730851    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-credential-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.743838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.736175    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-data\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.743838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.736822    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istiod-ca-cert\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.743838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.737384    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-workload-certs\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.743838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.737625    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-workload-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.784620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.784582    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j42hv\" (UniqueName: \"kubernetes.io/projected/0d6a7797-f46f-45bd-a2ca-47736b909cf9-kube-api-access-j42hv\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.785388 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.785366    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-token\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.804253 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.804166    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-envoy\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:08.812385 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:08.812129    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-podinfo\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:09.104391 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:09.104299    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:09.455560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:09.455526    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"]
Oct 29 22:35:09.460146 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:09.460096    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:09.460250 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:09.460196    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:09.460317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:09.460249    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:10.428837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:10.428587    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq" event={"ID":"0d6a7797-f46f-45bd-a2ca-47736b909cf9","Type":"ContainerStarted","Data":"c4a5219786d6befd7b16d47848679d84df82024bb6f167e1c9a8ba8b1339b75b"}
Oct 29 22:35:10.428837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:10.428644    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq" event={"ID":"0d6a7797-f46f-45bd-a2ca-47736b909cf9","Type":"ContainerStarted","Data":"76e43c18770c3528c806096308fdb4251239b630ec7e574da819935454ec4896"}
Oct 29 22:35:10.498023 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:10.497965    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq" podStartSLOduration=2.4979443740000002 podStartE2EDuration="2.497944374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:10.483042179 +0000 UTC m=+1249.394485733" watchObservedRunningTime="2026-10-29 22:35:10.497944374 +0000 UTC m=+1249.409387928"
Oct 29 22:35:11.105405 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:11.105373    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:11.110480 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:11.110457    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:11.437262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:11.437229    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:11.438192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:11.438172    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:35:12.313974 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:12.313947    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 22:35:13.695923 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:35:13.695884    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:35:21.440287 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:21.440255    4559 scope.go:122] "RemoveContainer" containerID="9b3128d1a077379af21676ea4392fc63a15c2c92d9fea4486fa725d1475b00e0"
Oct 29 22:35:21.453502 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:21.453476    4559 scope.go:122] "RemoveContainer" containerID="15467d342b814c7fc3bd5c1a01577d17983e43b5fbdb34a76d7e5bbdd12d6d4f"
Oct 29 22:35:24.045869 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:24.044880    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"]
Oct 29 22:35:24.050769 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:24.048539    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dea14161-shtcb"]
Oct 29 22:35:25.231237 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.231199    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1462bbfd-3301-4194-87ce-2961af36892a" path="/var/lib/kubelet/pods/1462bbfd-3301-4194-87ce-2961af36892a/volumes"
Oct 29 22:35:25.568656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.568567    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"]
Oct 29 22:35:25.576180 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.576157    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.597923 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.597891    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"]
Oct 29 22:35:25.603600 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.603570    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-3bd0f6-openshift-default-dockercfg-cmm6g\""
Oct 29 22:35:25.659501 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.659464    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fq72\" (UniqueName: \"kubernetes.io/projected/aa557804-9bd1-4a83-a858-1d150a4ac0fe-kube-api-access-8fq72\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.659689 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.659545    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-data\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.659689 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.659626    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-token\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.659689 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.659670    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istiod-ca-cert\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.659902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.659792    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-workload-certs\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.659902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.659877    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-credential-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.660135 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.660108    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-workload-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.660208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.660188    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-podinfo\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.660262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.660246    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-envoy\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.761465 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.761423    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-data\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.761646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.761472    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-token\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.761646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.761502    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istiod-ca-cert\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.761646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.761558    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-workload-certs\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.761646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.761599    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-credential-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.761906 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.761867    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-workload-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.761966 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.761936    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-podinfo\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.762019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.761965    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-data\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.762019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.761994    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-envoy\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.762117 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.762097    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fq72\" (UniqueName: \"kubernetes.io/projected/aa557804-9bd1-4a83-a858-1d150a4ac0fe-kube-api-access-8fq72\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.762117 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.762105    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-credential-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.762254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.762235    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-workload-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.762326 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.762312    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-workload-certs\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.762517 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.762426    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istiod-ca-cert\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.764738 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.764706    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-envoy\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.764988 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.764970    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-podinfo\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.781596 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.781561    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fq72\" (UniqueName: \"kubernetes.io/projected/aa557804-9bd1-4a83-a858-1d150a4ac0fe-kube-api-access-8fq72\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.781888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.781866    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-token\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:25.913182 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:25.913141    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:26.586963 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:35:26.586931    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa557804_9bd1_4a83_a858_1d150a4ac0fe.slice/crio-14618c2c4d2e89e8ee60da9acc09a4ec6971c687d308b57985bf4dd6d9861480 WatchSource:0}: Error finding container 14618c2c4d2e89e8ee60da9acc09a4ec6971c687d308b57985bf4dd6d9861480: Status 404 returned error can't find the container with id 14618c2c4d2e89e8ee60da9acc09a4ec6971c687d308b57985bf4dd6d9861480
Oct 29 22:35:26.587377 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:26.587168    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"]
Oct 29 22:35:26.589679 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:26.589643    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:26.589796 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:26.589721    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:26.589796 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:26.589762    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:26.661490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:26.661446    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh" event={"ID":"aa557804-9bd1-4a83-a858-1d150a4ac0fe","Type":"ContainerStarted","Data":"433dae1e18022d12eed9c784849ede4540a8087123ec41a216b1089dc8d76f7d"}
Oct 29 22:35:26.661700 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:26.661519    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh" event={"ID":"aa557804-9bd1-4a83-a858-1d150a4ac0fe","Type":"ContainerStarted","Data":"14618c2c4d2e89e8ee60da9acc09a4ec6971c687d308b57985bf4dd6d9861480"}
Oct 29 22:35:26.705639 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:26.705573    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh" podStartSLOduration=1.705557207 podStartE2EDuration="1.705557207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:26.703135382 +0000 UTC m=+1265.614578941" watchObservedRunningTime="2026-10-29 22:35:26.705557207 +0000 UTC m=+1265.617000797"
Oct 29 22:35:26.916900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:26.915307    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:27.923778 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:27.923138    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:28.680486 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:28.680452    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:28.681775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:28.681755    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:35:29.524763 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:29.524724    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"]
Oct 29 22:35:29.529483 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:29.529458    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:35:29.569834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:29.557883    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3255ca0c-f905-4eae-bb1b-b4d06622086d-metrics-certs\") pod \"maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf\" (UID: \"3255ca0c-f905-4eae-bb1b-b4d06622086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:35:29.569834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:29.558920    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp5l6\" (UniqueName: \"kubernetes.io/projected/3255ca0c-f905-4eae-bb1b-b4d06622086d-kube-api-access-cp5l6\") pod \"maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf\" (UID: \"3255ca0c-f905-4eae-bb1b-b4d06622086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:35:29.569834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:29.559175    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3255ca0c-f905-4eae-bb1b-b4d06622086d-maas-api-tls\") pod \"maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf\" (UID: \"3255ca0c-f905-4eae-bb1b-b4d06622086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:35:29.569834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:29.559242    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3255ca0c-f905-4eae-bb1b-b4d06622086d-openshift-service-ca\") pod \"maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf\" (UID: \"3255ca0c-f905-4eae-bb1b-b4d06622086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:35:29.577850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:29.570427    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1131a29d-serving-cert\""
Oct 29 22:35:29.578080 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:29.573974    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"]
Oct 29 22:35:29.660902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:29.660857    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3255ca0c-f905-4eae-bb1b-b4d06622086d-metrics-certs\") pod \"maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf\" (UID: \"3255ca0c-f905-4eae-bb1b-b4d06622086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:35:29.661111 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:29.660918    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp5l6\" (UniqueName: \"kubernetes.io/projected/3255ca0c-f905-4eae-bb1b-b4d06622086d-kube-api-access-cp5l6\") pod \"maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf\" (UID: \"3255ca0c-f905-4eae-bb1b-b4d06622086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:35:29.661186 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:29.661149    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3255ca0c-f905-4eae-bb1b-b4d06622086d-maas-api-tls\") pod \"maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf\" (UID: \"3255ca0c-f905-4eae-bb1b-b4d06622086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:35:29.661255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:29.661214    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3255ca0c-f905-4eae-bb1b-b4d06622086d-openshift-service-ca\") pod \"maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf\" (UID: \"3255ca0c-f905-4eae-bb1b-b4d06622086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:35:29.662369 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:29.662315    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3255ca0c-f905-4eae-bb1b-b4d06622086d-openshift-service-ca\") pod \"maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf\" (UID: \"3255ca0c-f905-4eae-bb1b-b4d06622086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:35:29.664029 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:29.663998    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3255ca0c-f905-4eae-bb1b-b4d06622086d-metrics-certs\") pod \"maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf\" (UID: \"3255ca0c-f905-4eae-bb1b-b4d06622086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:35:29.664233 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:29.664214    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3255ca0c-f905-4eae-bb1b-b4d06622086d-maas-api-tls\") pod \"maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf\" (UID: \"3255ca0c-f905-4eae-bb1b-b4d06622086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:35:29.682514 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:29.682481    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp5l6\" (UniqueName: \"kubernetes.io/projected/3255ca0c-f905-4eae-bb1b-b4d06622086d-kube-api-access-cp5l6\") pod \"maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf\" (UID: \"3255ca0c-f905-4eae-bb1b-b4d06622086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:35:29.846894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:29.846194    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:35:30.159678 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:30.159629    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:35:30.406881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:30.402156    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"]
Oct 29 22:35:30.696154 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:30.696063    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf" event={"ID":"3255ca0c-f905-4eae-bb1b-b4d06622086d","Type":"ContainerStarted","Data":"7b0756a17418c6b0b8841615b8e17a64cba39f34a93d155364e56f7e52a77532"}
Oct 29 22:35:31.802834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:31.802374    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf" event={"ID":"3255ca0c-f905-4eae-bb1b-b4d06622086d","Type":"ContainerStarted","Data":"772784b523a66d07bbc64e105b5840ea6595bf2c6b368c73419ab8a737c7a5e2"}
Oct 29 22:35:31.814837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:31.808728    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:35:32.013488 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:32.013365    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf" podStartSLOduration=2.588403736 podStartE2EDuration="3.013343802s" totalImagesPullingTime="424.940066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:31.884377886 +0000 UTC m=+1270.795821453" watchObservedRunningTime="2026-10-29 22:35:32.013343802 +0000 UTC m=+1270.924787378"
Oct 29 22:35:36.492376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.492323    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"]
Oct 29 22:35:36.517667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.516288    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.711839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.671660    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.711839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.671832    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.711839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.671861    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldxwz\" (UniqueName: \"kubernetes.io/projected/cc440498-ac20-43de-b45a-3d0bcb2c312f-kube-api-access-ldxwz\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.711839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.671910    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.711839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.672001    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.711839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.672047    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc440498-ac20-43de-b45a-3d0bcb2c312f-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.711839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.672084    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.711839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.672130    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.711839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.672164    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.711839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.694350    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"]
Oct 29 22:35:36.814922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.797918    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.814922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.798004    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc440498-ac20-43de-b45a-3d0bcb2c312f-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.814922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.798048    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.814922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.798106    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.814922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.798149    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.814922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.798273    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.814922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.798430    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.814922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.798463    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldxwz\" (UniqueName: \"kubernetes.io/projected/cc440498-ac20-43de-b45a-3d0bcb2c312f-kube-api-access-ldxwz\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.814922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.798525    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.814922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.799177    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.814922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.807212    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc440498-ac20-43de-b45a-3d0bcb2c312f-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.814922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.808033    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.814922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.808257    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.814922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.809009    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.851857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.851799    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-m55mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:35:36.852135 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.852092    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt" podUID="d166a89c-98b6-4209-b8d8-5b405090031b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:35:36.919201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.919138    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.919634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.919604    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.920370 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.920339    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:36.920910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:36.920890    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldxwz\" (UniqueName: \"kubernetes.io/projected/cc440498-ac20-43de-b45a-3d0bcb2c312f-kube-api-access-ldxwz\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:37.106972 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:37.106869    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:37.609107 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:35:37.608695    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc440498_ac20_43de_b45a_3d0bcb2c312f.slice/crio-7ae0217528fc741a5bec7f89d9f8031f6759889a6f4af3f9a2603a66262f59d5 WatchSource:0}: Error finding container 7ae0217528fc741a5bec7f89d9f8031f6759889a6f4af3f9a2603a66262f59d5: Status 404 returned error can't find the container with id 7ae0217528fc741a5bec7f89d9f8031f6759889a6f4af3f9a2603a66262f59d5
Oct 29 22:35:37.611339 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:37.611312    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"]
Oct 29 22:35:37.612268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:37.612239    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:37.612342 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:37.612328    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:37.612381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:37.612374    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:38.019842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:38.019782    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8" event={"ID":"cc440498-ac20-43de-b45a-3d0bcb2c312f","Type":"ContainerStarted","Data":"6da36e66fcb64f5c4d6845f8ddf2301f7c2d1110c6a44f7b75a62218090d4011"}
Oct 29 22:35:38.019997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:38.019867    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8" event={"ID":"cc440498-ac20-43de-b45a-3d0bcb2c312f","Type":"ContainerStarted","Data":"7ae0217528fc741a5bec7f89d9f8031f6759889a6f4af3f9a2603a66262f59d5"}
Oct 29 22:35:38.111983 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:38.111858    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:38.112126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:38.112045    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:35:38.112126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:38.112100    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8" podUID="cc440498-ac20-43de-b45a-3d0bcb2c312f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 29 22:35:38.115548 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:38.114047    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8" podStartSLOduration=2.114034487 podStartE2EDuration="2.114034487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:38.108488269 +0000 UTC m=+1277.019931835" watchObservedRunningTime="2026-10-29 22:35:38.114034487 +0000 UTC m=+1277.025478106"
Oct 29 22:35:38.884569 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:38.884495    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:35:39.149902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:39.149030    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:35:39.149902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:39.149099    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8" podUID="cc440498-ac20-43de-b45a-3d0bcb2c312f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 29 22:35:40.108167 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:40.108126    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:35:40.108647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:40.108185    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8" podUID="cc440498-ac20-43de-b45a-3d0bcb2c312f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 29 22:35:41.108419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:41.108385    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:35:41.108920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:41.108443    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8" podUID="cc440498-ac20-43de-b45a-3d0bcb2c312f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 29 22:35:42.111847 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:42.108637    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:35:42.111847 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:42.108697    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8" podUID="cc440498-ac20-43de-b45a-3d0bcb2c312f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 29 22:35:43.108005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:43.107974    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:35:43.108286 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:43.108263    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8" podUID="cc440498-ac20-43de-b45a-3d0bcb2c312f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 29 22:35:43.532831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:43.532770    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"]
Oct 29 22:35:43.575834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:43.572231    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:35:43.595877 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:43.595844    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"]
Oct 29 22:35:43.603344 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:43.602965    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-db7419-serving-cert\""
Oct 29 22:35:43.697712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:43.695900    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43b52f1-aade-4064-9d8a-111d155c659b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9\" (UID: \"b43b52f1-aade-4064-9d8a-111d155c659b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:35:43.697712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:43.696180    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzcgx\" (UniqueName: \"kubernetes.io/projected/b43b52f1-aade-4064-9d8a-111d155c659b-kube-api-access-hzcgx\") pod \"maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9\" (UID: \"b43b52f1-aade-4064-9d8a-111d155c659b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:35:43.697712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:43.696333    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b43b52f1-aade-4064-9d8a-111d155c659b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9\" (UID: \"b43b52f1-aade-4064-9d8a-111d155c659b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:35:43.697712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:43.696472    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b43b52f1-aade-4064-9d8a-111d155c659b-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9\" (UID: \"b43b52f1-aade-4064-9d8a-111d155c659b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:35:43.827975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:43.825352    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b43b52f1-aade-4064-9d8a-111d155c659b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9\" (UID: \"b43b52f1-aade-4064-9d8a-111d155c659b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:35:43.827975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:43.825523    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b43b52f1-aade-4064-9d8a-111d155c659b-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9\" (UID: \"b43b52f1-aade-4064-9d8a-111d155c659b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:35:43.827975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:43.825625    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43b52f1-aade-4064-9d8a-111d155c659b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9\" (UID: \"b43b52f1-aade-4064-9d8a-111d155c659b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:35:43.827975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:43.825828    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzcgx\" (UniqueName: \"kubernetes.io/projected/b43b52f1-aade-4064-9d8a-111d155c659b-kube-api-access-hzcgx\") pod \"maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9\" (UID: \"b43b52f1-aade-4064-9d8a-111d155c659b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:35:43.827975 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:35:43.826329    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-serving-cert: secret "maas-api-e2e-shared-b-w6-db7419-serving-cert" not found
Oct 29 22:35:43.827975 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:35:43.826435    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43b52f1-aade-4064-9d8a-111d155c659b-maas-api-tls podName:b43b52f1-aade-4064-9d8a-111d155c659b nodeName:}" failed. No retries permitted until 2026-10-29 22:35:44.326408118 +0000 UTC m=+1283.237851651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b43b52f1-aade-4064-9d8a-111d155c659b-maas-api-tls") pod "maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9" (UID: "b43b52f1-aade-4064-9d8a-111d155c659b") : secret "maas-api-e2e-shared-b-w6-db7419-serving-cert" not found
Oct 29 22:35:43.827975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:43.827869    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43b52f1-aade-4064-9d8a-111d155c659b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9\" (UID: \"b43b52f1-aade-4064-9d8a-111d155c659b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:35:44.008938 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:44.008899    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzcgx\" (UniqueName: \"kubernetes.io/projected/b43b52f1-aade-4064-9d8a-111d155c659b-kube-api-access-hzcgx\") pod \"maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9\" (UID: \"b43b52f1-aade-4064-9d8a-111d155c659b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:35:44.011369 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:35:44.011306    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:35:44.015229 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:44.015160    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b43b52f1-aade-4064-9d8a-111d155c659b-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9\" (UID: \"b43b52f1-aade-4064-9d8a-111d155c659b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:35:44.108585 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:44.107695    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:35:44.108585 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:44.107768    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8" podUID="cc440498-ac20-43de-b45a-3d0bcb2c312f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 29 22:35:44.377093 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:44.376977    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b43b52f1-aade-4064-9d8a-111d155c659b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9\" (UID: \"b43b52f1-aade-4064-9d8a-111d155c659b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:35:44.429766 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:44.429508    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b43b52f1-aade-4064-9d8a-111d155c659b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9\" (UID: \"b43b52f1-aade-4064-9d8a-111d155c659b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:35:44.605365 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:44.605322    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:35:45.096717 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:45.096588    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"]
Oct 29 22:35:45.128844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:45.118704    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:35:45.128844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:45.118769    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8" podUID="cc440498-ac20-43de-b45a-3d0bcb2c312f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 29 22:35:45.358676 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:45.358583    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9" event={"ID":"b43b52f1-aade-4064-9d8a-111d155c659b","Type":"ContainerStarted","Data":"262a22edc6c2f2156a131dbec3239f1c6535f66ec52c5dd51917a28bd66c1391"}
Oct 29 22:35:46.108252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:46.108213    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:35:46.108779 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:46.108285    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8" podUID="cc440498-ac20-43de-b45a-3d0bcb2c312f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 29 22:35:46.373879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:46.373719    4559 generic.go:356] "Generic (PLEG): container finished" podID="f479373b-2190-46af-9b6d-ad901981a56b" containerID="24dfc707efe15a6ea6c7fb90c9a6b6461ceccf29b006803f913d700c006b1cf7" exitCode=0
Oct 29 22:35:46.374055 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:46.373885    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m" event={"ID":"f479373b-2190-46af-9b6d-ad901981a56b","Type":"ContainerDied","Data":"24dfc707efe15a6ea6c7fb90c9a6b6461ceccf29b006803f913d700c006b1cf7"}
Oct 29 22:35:46.374543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:46.374523    4559 scope.go:122] "RemoveContainer" containerID="24dfc707efe15a6ea6c7fb90c9a6b6461ceccf29b006803f913d700c006b1cf7"
Oct 29 22:35:46.376491 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:46.376392    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9" event={"ID":"b43b52f1-aade-4064-9d8a-111d155c659b","Type":"ContainerStarted","Data":"19a04cc987a4b00abeb9015755d47c906f1b1925c4139b18b79758007e0212af"}
Oct 29 22:35:46.377070 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:46.377053    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:35:46.377942 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:46.377909    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:46.378019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:46.377997    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:46.378061 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:46.378041    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:46.512209 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:46.512147    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9" podStartSLOduration=3.005889988 podStartE2EDuration="3.512130681s" totalImagesPullingTime="506.240693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:46.482095719 +0000 UTC m=+1285.393539296" watchObservedRunningTime="2026-10-29 22:35:46.512130681 +0000 UTC m=+1285.423574235"
Oct 29 22:35:47.161218 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.161185    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:47.168672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.162281    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:47.173661 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.173589    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:35:47.390381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.390349    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m" event={"ID":"f479373b-2190-46af-9b6d-ad901981a56b","Type":"ContainerStarted","Data":"721665c9e550ef6adbdbf3f4bab54654b871141ed3e1b22105189711c92247c7"}
Oct 29 22:35:47.394560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.392971    4559 generic.go:356] "Generic (PLEG): container finished" podID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" containerID="ee567f4816a59d997f20a6010840ceabcf7bd1ac8e50bb6739a48263fb292279" exitCode=0
Oct 29 22:35:47.394560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.393641    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh" event={"ID":"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c","Type":"ContainerDied","Data":"ee567f4816a59d997f20a6010840ceabcf7bd1ac8e50bb6739a48263fb292279"}
Oct 29 22:35:47.394560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.393957    4559 scope.go:122] "RemoveContainer" containerID="ee567f4816a59d997f20a6010840ceabcf7bd1ac8e50bb6739a48263fb292279"
Oct 29 22:35:47.395911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.395871    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:47.396041 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.395966    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:47.396041 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.396009    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:47.590841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.589294    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"]
Oct 29 22:35:47.599660 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.597140    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"
Oct 29 22:35:47.619665 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.619639    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1fa4b8b6\""
Oct 29 22:35:47.620133 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.620110    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1fa4b8b6-dockercfg-s45m4\""
Oct 29 22:35:47.675522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.675417    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"]
Oct 29 22:35:47.686864 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.683756    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"]
Oct 29 22:35:47.690988 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.690093    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:35:47.690988 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.690214    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"
Oct 29 22:35:47.704238 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.704080    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e695e40-8026-49fb-80f3-f8c22dcf206d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw\" (UID: \"6e695e40-8026-49fb-80f3-f8c22dcf206d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"
Oct 29 22:35:47.704388 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.704270    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k6x4\" (UniqueName: \"kubernetes.io/projected/6e695e40-8026-49fb-80f3-f8c22dcf206d-kube-api-access-4k6x4\") pod \"payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw\" (UID: \"6e695e40-8026-49fb-80f3-f8c22dcf206d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"
Oct 29 22:35:47.723982 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.723948    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"]
Oct 29 22:35:47.818651 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.815919    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg6pp\" (UniqueName: \"kubernetes.io/projected/11c6a530-568d-42fa-a268-344b4fb7bd19-kube-api-access-vg6pp\") pod \"payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869\" (UID: \"11c6a530-568d-42fa-a268-344b4fb7bd19\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"
Oct 29 22:35:47.818651 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.816218    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e695e40-8026-49fb-80f3-f8c22dcf206d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw\" (UID: \"6e695e40-8026-49fb-80f3-f8c22dcf206d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"
Oct 29 22:35:47.818651 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.816369    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c6a530-568d-42fa-a268-344b4fb7bd19-plugins-config-volume\") pod \"payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869\" (UID: \"11c6a530-568d-42fa-a268-344b4fb7bd19\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"
Oct 29 22:35:47.818651 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.816423    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k6x4\" (UniqueName: \"kubernetes.io/projected/6e695e40-8026-49fb-80f3-f8c22dcf206d-kube-api-access-4k6x4\") pod \"payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw\" (UID: \"6e695e40-8026-49fb-80f3-f8c22dcf206d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"
Oct 29 22:35:47.818651 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.817787    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e695e40-8026-49fb-80f3-f8c22dcf206d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw\" (UID: \"6e695e40-8026-49fb-80f3-f8c22dcf206d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"
Oct 29 22:35:47.941686 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.939943    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg6pp\" (UniqueName: \"kubernetes.io/projected/11c6a530-568d-42fa-a268-344b4fb7bd19-kube-api-access-vg6pp\") pod \"payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869\" (UID: \"11c6a530-568d-42fa-a268-344b4fb7bd19\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"
Oct 29 22:35:47.941686 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.940321    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c6a530-568d-42fa-a268-344b4fb7bd19-plugins-config-volume\") pod \"payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869\" (UID: \"11c6a530-568d-42fa-a268-344b4fb7bd19\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"
Oct 29 22:35:47.941686 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.941284    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c6a530-568d-42fa-a268-344b4fb7bd19-plugins-config-volume\") pod \"payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869\" (UID: \"11c6a530-568d-42fa-a268-344b4fb7bd19\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"
Oct 29 22:35:48.018835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:47.996549    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg6pp\" (UniqueName: \"kubernetes.io/projected/11c6a530-568d-42fa-a268-344b4fb7bd19-kube-api-access-vg6pp\") pod \"payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869\" (UID: \"11c6a530-568d-42fa-a268-344b4fb7bd19\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"
Oct 29 22:35:48.088836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:48.079088    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"
Oct 29 22:35:48.121917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:48.121686    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k6x4\" (UniqueName: \"kubernetes.io/projected/6e695e40-8026-49fb-80f3-f8c22dcf206d-kube-api-access-4k6x4\") pod \"payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw\" (UID: \"6e695e40-8026-49fb-80f3-f8c22dcf206d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"
Oct 29 22:35:48.220219 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:48.220137    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"
Oct 29 22:35:48.313435 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:48.313402    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:35:48.314826 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:48.314783    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:35:48.314951 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:48.314851    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m" podUID="f479373b-2190-46af-9b6d-ad901981a56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 22:35:48.430190 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:48.430151    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh" event={"ID":"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c","Type":"ContainerStarted","Data":"bf730e11dd715bae70d568b0c1bb8b539302e665031e0e919f618f7a0bc79880"}
Oct 29 22:35:48.695834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:48.693415    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"]
Oct 29 22:35:48.698798 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:48.698513    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"]
Oct 29 22:35:49.028629 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:49.028533    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:35:49.029317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:49.029290    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:35:49.029444 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:49.029344    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh" podUID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:35:49.321917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:49.313239    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:35:49.321917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:49.313300    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m" podUID="f479373b-2190-46af-9b6d-ad901981a56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 22:35:49.546836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:49.546012    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw" event={"ID":"6e695e40-8026-49fb-80f3-f8c22dcf206d","Type":"ContainerStarted","Data":"d0e5c135f00706ba888fa57aaa81413b319a1bed968943d91039ba57190a3cbd"}
Oct 29 22:35:49.546836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:49.546101    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw" event={"ID":"6e695e40-8026-49fb-80f3-f8c22dcf206d","Type":"ContainerStarted","Data":"8b0a33856c189278a29348a8d03041ee4605d051bb04d4cf924ea33553e6dd69"}
Oct 29 22:35:49.550838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:49.547404    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"
Oct 29 22:35:49.591931 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:49.589084    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869" event={"ID":"11c6a530-568d-42fa-a268-344b4fb7bd19","Type":"ContainerStarted","Data":"0122ca2b3add8cafbf49e57678f10eb8cceb7bd2d436e2cce033f4975b28f423"}
Oct 29 22:35:49.591931 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:49.589144    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869" event={"ID":"11c6a530-568d-42fa-a268-344b4fb7bd19","Type":"ContainerStarted","Data":"a49760b90759cc000e9ee7f9d3fe1a14046b9e22c1f7ad7aa2778b0792e992b6"}
Oct 29 22:35:49.591931 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:49.589174    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"
Oct 29 22:35:49.604308 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:49.604234    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw" podStartSLOduration=2.604201968 podStartE2EDuration="2.604201968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:49.598706705 +0000 UTC m=+1288.510150260" watchObservedRunningTime="2026-10-29 22:35:49.604201968 +0000 UTC m=+1288.515645523"
Oct 29 22:35:49.699564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:49.699483    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869" podStartSLOduration=2.699461714 podStartE2EDuration="2.699461714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:49.667158411 +0000 UTC m=+1288.578601966" watchObservedRunningTime="2026-10-29 22:35:49.699461714 +0000 UTC m=+1288.610905270"
Oct 29 22:35:50.028767 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:50.028730    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:35:50.028972 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:50.028784    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh" podUID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:35:50.313713 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:50.313625    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:35:50.313713 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:50.313682    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m" podUID="f479373b-2190-46af-9b6d-ad901981a56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 22:35:51.042636 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:51.038266    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:35:51.042636 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:51.038338    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh" podUID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:35:51.313584 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:51.313412    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:35:51.313584 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:51.313477    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m" podUID="f479373b-2190-46af-9b6d-ad901981a56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 22:35:52.036837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:52.032482    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:35:52.036837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:52.032646    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh" podUID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:35:52.317340 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:52.317252    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:35:52.317939 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:52.317904    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m" podUID="f479373b-2190-46af-9b6d-ad901981a56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 22:35:52.744840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:52.740977    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:35:53.029257 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:53.029163    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:35:53.029257 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:53.029236    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh" podUID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:35:53.313448 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:53.313353    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:35:53.313448 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:53.313420    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m" podUID="f479373b-2190-46af-9b6d-ad901981a56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 22:35:53.417843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:53.417783    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:35:54.056126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:54.056085    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:35:54.056429 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:54.056401    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh" podUID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:35:54.425846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:54.416150    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:35:54.521950 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:54.521920    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 22:35:55.029273 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:55.029240    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:35:55.029460 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:55.029292    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh" podUID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:35:56.121746 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:56.121709    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:35:56.124069 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:56.124041    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:35:58.833112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.833062    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"]
Oct 29 22:35:58.842987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.842959    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"]
Oct 29 22:35:58.850213 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.843074    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:58.851049 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.851026    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:58.854227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.854193    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"]
Oct 29 22:35:58.875163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.875123    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"]
Oct 29 22:35:58.984420 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.984361    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:58.984633 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.984457    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:58.984633 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.984525    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:58.984744 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.984638    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:58.984744 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.984673    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:58.984744 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.984698    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:58.984922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.984753    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:58.984922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.984791    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:58.984993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.984925    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:58.984993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.984981    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:58.985056 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.985025    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:58.985120 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.985107    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:58.985174 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.985164    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5nn2\" (UniqueName: \"kubernetes.io/projected/09ca3404-6dc6-409a-9eab-8bf1c44cc045-kube-api-access-w5nn2\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:58.985212 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.985183    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:58.985296 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.985277    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:58.985349 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.985332    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:58.985402 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.985387    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzzqc\" (UniqueName: \"kubernetes.io/projected/ec0ef457-75e7-432c-8023-3ef04b7dbbda-kube-api-access-pzzqc\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:58.985460 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:58.985449    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.087240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.087145    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.087240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.087195    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.087473 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.087268    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.087473 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.087336    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.087473 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.087368    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.087879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.087862    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.087968 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.087604    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.087968 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.087900    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.087968 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.087795    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.087968 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.087662    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.087968 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.087802    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.088184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.087977    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.088184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.088045    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.088184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.088074    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.088184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.088110    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.088184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.088183    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.088442 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.088269    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5nn2\" (UniqueName: \"kubernetes.io/projected/09ca3404-6dc6-409a-9eab-8bf1c44cc045-kube-api-access-w5nn2\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.088442 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.088295    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.088442 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.088343    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.088442 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.088347    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.088442 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.088415    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.088741 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.088454    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzzqc\" (UniqueName: \"kubernetes.io/projected/ec0ef457-75e7-432c-8023-3ef04b7dbbda-kube-api-access-pzzqc\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.088741 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.088506    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.088741 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.088591    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.089002 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.088794    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.089002 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.088833    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.089002 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.088864    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.089738 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.089710    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.090802 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.090760    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.090939 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.090920    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.091325 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.091299    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.091491 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.091462    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.108936 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.108901    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.109215 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.109193    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.109428 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.109408    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5nn2\" (UniqueName: \"kubernetes.io/projected/09ca3404-6dc6-409a-9eab-8bf1c44cc045-kube-api-access-w5nn2\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r8hqd\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.109496 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.109411    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzzqc\" (UniqueName: \"kubernetes.io/projected/ec0ef457-75e7-432c-8023-3ef04b7dbbda-kube-api-access-pzzqc\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-z7hjw\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.197347 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.197310    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:35:59.199272 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.199177    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:35:59.667401 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:35:59.667369    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ca3404_6dc6_409a_9eab_8bf1c44cc045.slice/crio-3697639271afe9c8e1c4af7b9e5be0f344ea70e1717186aa300f53a7a4b1a004 WatchSource:0}: Error finding container 3697639271afe9c8e1c4af7b9e5be0f344ea70e1717186aa300f53a7a4b1a004: Status 404 returned error can't find the container with id 3697639271afe9c8e1c4af7b9e5be0f344ea70e1717186aa300f53a7a4b1a004
Oct 29 22:35:59.668917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.668893    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"]
Oct 29 22:35:59.670753 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.670719    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:59.670880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.670788    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:59.670880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.670838    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:59.680308 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.680283    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"]
Oct 29 22:35:59.684597 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:35:59.684571    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0ef457_75e7_432c_8023_3ef04b7dbbda.slice/crio-50279129eb40d3222d57c0e8f83b055d14ca535ff262976f64b1bab7319c9313 WatchSource:0}: Error finding container 50279129eb40d3222d57c0e8f83b055d14ca535ff262976f64b1bab7319c9313: Status 404 returned error can't find the container with id 50279129eb40d3222d57c0e8f83b055d14ca535ff262976f64b1bab7319c9313
Oct 29 22:35:59.687416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.687382    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:59.687518 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.687462    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:59.687597 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.687507    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:35:59.895831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.894933    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw" event={"ID":"ec0ef457-75e7-432c-8023-3ef04b7dbbda","Type":"ContainerStarted","Data":"bc0c118ac6eeda6aa744213e8179aec762cf5103094ca057bfa5f2be213c3206"}
Oct 29 22:35:59.895831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.894997    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw" event={"ID":"ec0ef457-75e7-432c-8023-3ef04b7dbbda","Type":"ContainerStarted","Data":"50279129eb40d3222d57c0e8f83b055d14ca535ff262976f64b1bab7319c9313"}
Oct 29 22:35:59.896893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.896858    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd" event={"ID":"09ca3404-6dc6-409a-9eab-8bf1c44cc045","Type":"ContainerStarted","Data":"993408cb4d70b3db27efd789268a6b1a4035a5dabbafea32320dabd76bfddac1"}
Oct 29 22:35:59.897044 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.896908    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd" event={"ID":"09ca3404-6dc6-409a-9eab-8bf1c44cc045","Type":"ContainerStarted","Data":"3697639271afe9c8e1c4af7b9e5be0f344ea70e1717186aa300f53a7a4b1a004"}
Oct 29 22:35:59.944609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.944409    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw" podStartSLOduration=1.9443863590000001 podStartE2EDuration="1.944386359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:59.941171433 +0000 UTC m=+1298.852615008" watchObservedRunningTime="2026-10-29 22:35:59.944386359 +0000 UTC m=+1298.855829913"
Oct 29 22:35:59.990530 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:35:59.989969    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd" podStartSLOduration=1.989949202 podStartE2EDuration="1.989949202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:59.989110359 +0000 UTC m=+1298.900553915" watchObservedRunningTime="2026-10-29 22:35:59.989949202 +0000 UTC m=+1298.901392756"
Oct 29 22:36:00.198542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:00.198439    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:36:00.199564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:00.199532    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:36:00.200639 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:00.200613    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:36:00.200773 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:00.200670    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd" podUID="09ca3404-6dc6-409a-9eab-8bf1c44cc045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:36:00.200929 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:00.200910    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:36:00.200990 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:00.200963    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw" podUID="ec0ef457-75e7-432c-8023-3ef04b7dbbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:36:00.607276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:00.607153    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"
Oct 29 22:36:01.198299 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:01.198245    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:36:01.198703 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:01.198321    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd" podUID="09ca3404-6dc6-409a-9eab-8bf1c44cc045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:36:01.200310 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:01.200283    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:36:01.200457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:01.200321    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw" podUID="ec0ef457-75e7-432c-8023-3ef04b7dbbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:36:01.685838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:01.685037    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"
Oct 29 22:36:02.199937 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:02.199239    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:36:02.199937 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:02.199306    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd" podUID="09ca3404-6dc6-409a-9eab-8bf1c44cc045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:36:02.200494 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:02.200093    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:36:02.200494 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:02.200143    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw" podUID="ec0ef457-75e7-432c-8023-3ef04b7dbbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:36:03.199836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:03.199163    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:36:03.199836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:03.199220    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd" podUID="09ca3404-6dc6-409a-9eab-8bf1c44cc045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:36:03.207916 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:03.201168    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:36:03.207916 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:03.201221    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw" podUID="ec0ef457-75e7-432c-8023-3ef04b7dbbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:36:04.200871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:04.199302    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:36:04.200871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:04.199387    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd" podUID="09ca3404-6dc6-409a-9eab-8bf1c44cc045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:36:04.200871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:04.200153    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:36:04.200871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:04.200213    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw" podUID="ec0ef457-75e7-432c-8023-3ef04b7dbbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:36:05.199507 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:05.198396    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:36:05.199507 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:05.198476    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd" podUID="09ca3404-6dc6-409a-9eab-8bf1c44cc045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:36:05.199790 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:05.199761    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:36:05.199878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:05.199832    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw" podUID="ec0ef457-75e7-432c-8023-3ef04b7dbbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:36:06.212250 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:06.212208    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:36:06.212790 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:06.212275    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw" podUID="ec0ef457-75e7-432c-8023-3ef04b7dbbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:36:06.213332 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:06.213302    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:36:06.213592 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:06.213556    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd" podUID="09ca3404-6dc6-409a-9eab-8bf1c44cc045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:36:07.198212 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:07.198172    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:36:07.198452 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:07.198234    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd" podUID="09ca3404-6dc6-409a-9eab-8bf1c44cc045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:36:07.200038 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:07.199989    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:36:07.200196 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:07.200053    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw" podUID="ec0ef457-75e7-432c-8023-3ef04b7dbbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:36:08.198149 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:08.198111    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:36:08.198583 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:08.198175    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd" podUID="09ca3404-6dc6-409a-9eab-8bf1c44cc045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 22:36:08.200548 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:08.200519    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:36:08.200641 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:08.200576    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw" podUID="ec0ef457-75e7-432c-8023-3ef04b7dbbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:36:09.206122 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:09.206079    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:36:09.206726 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:09.206691    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw" podUID="ec0ef457-75e7-432c-8023-3ef04b7dbbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:36:10.224839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:10.207149    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:36:10.224839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:10.207214    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw" podUID="ec0ef457-75e7-432c-8023-3ef04b7dbbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 29 22:36:10.246057 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:10.246013    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:36:10.246257 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:10.246095    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd" podUID="09ca3404-6dc6-409a-9eab-8bf1c44cc045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:36:10.688867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:10.688012    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:36:10.689156 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:10.689131    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:36:10.690908 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:10.690876    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:36:12.116838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:12.116111    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:36:12.127839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:12.117909    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:36:12.155833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:12.155119    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:36:13.787828 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:13.787653    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:36:20.716910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:20.716872    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9092-09835a"]
Oct 29 22:36:20.722729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:20.722704    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-09835a"
Oct 29 22:36:20.772867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:20.757127    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xl57t\""
Oct 29 22:36:20.777938 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:20.777902    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9092-09835a"]
Oct 29 22:36:20.778801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:20.778758    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w499g\" (UniqueName: \"kubernetes.io/projected/c9899b51-add6-4929-b236-5bd6a1af5257-kube-api-access-w499g\") pod \"test-curl-9092-09835a\" (UID: \"c9899b51-add6-4929-b236-5bd6a1af5257\") " pod="opendatahub/test-curl-9092-09835a"
Oct 29 22:36:20.880226 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:20.880192    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w499g\" (UniqueName: \"kubernetes.io/projected/c9899b51-add6-4929-b236-5bd6a1af5257-kube-api-access-w499g\") pod \"test-curl-9092-09835a\" (UID: \"c9899b51-add6-4929-b236-5bd6a1af5257\") " pod="opendatahub/test-curl-9092-09835a"
Oct 29 22:36:20.900189 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:20.900152    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w499g\" (UniqueName: \"kubernetes.io/projected/c9899b51-add6-4929-b236-5bd6a1af5257-kube-api-access-w499g\") pod \"test-curl-9092-09835a\" (UID: \"c9899b51-add6-4929-b236-5bd6a1af5257\") " pod="opendatahub/test-curl-9092-09835a"
Oct 29 22:36:21.094702 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:21.094610    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-09835a"
Oct 29 22:36:21.506896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:21.504892    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9092-09835a"]
Oct 29 22:36:21.801388 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:21.801310    4559 scope.go:122] "RemoveContainer" containerID="f4bdcb67fe7e600088f0d4a3431669694fd029436f4f7ce9c7e5320ed385b749"
Oct 29 22:36:21.872833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:21.872779    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9092-09835a" event={"ID":"c9899b51-add6-4929-b236-5bd6a1af5257","Type":"ContainerStarted","Data":"3414d33dac71599e94542508b94fd30020d49ac87d35ac0beee5c1cfd87a4e33"}
Oct 29 22:36:24.904053 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:24.904007    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9092-09835a" event={"ID":"c9899b51-add6-4929-b236-5bd6a1af5257","Type":"ContainerStarted","Data":"d555300eeebc0396753bc2ba4c1c67db5d34eb332d1fb8b430ffc987c17f6879"}
Oct 29 22:36:24.945746 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:24.945677    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9092-09835a" podStartSLOduration=2.219793858 podStartE2EDuration="4.945654163s" totalImagesPullingTime="2.725860305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:24.940668469 +0000 UTC m=+1323.852112023" watchObservedRunningTime="2026-10-29 22:36:24.945654163 +0000 UTC m=+1323.857097719"
Oct 29 22:36:25.788792 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:25.788753    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9092-09835a"]
Oct 29 22:36:25.800626 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:25.800592    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9092-09835a"]
Oct 29 22:36:25.950532 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:25.950494    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9092-10c45a"]
Oct 29 22:36:25.951306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:25.951289    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c9899b51-add6-4929-b236-5bd6a1af5257" containerName="test-curl-9092-09835a"
Oct 29 22:36:25.951349 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:25.951306    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9899b51-add6-4929-b236-5bd6a1af5257" containerName="test-curl-9092-09835a"
Oct 29 22:36:25.951421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:25.951412    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9899b51-add6-4929-b236-5bd6a1af5257" containerName="test-curl-9092-09835a"
Oct 29 22:36:25.955306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:25.955287    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-10c45a"
Oct 29 22:36:25.970991 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:25.970962    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9092-10c45a"]
Oct 29 22:36:25.972372 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:25.971635    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc5d4\" (UniqueName: \"kubernetes.io/projected/a848f387-f33e-4c4d-be5b-30757ea54810-kube-api-access-vc5d4\") pod \"test-curl-9092-10c45a\" (UID: \"a848f387-f33e-4c4d-be5b-30757ea54810\") " pod="opendatahub/test-curl-9092-10c45a"
Oct 29 22:36:26.073752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:26.073658    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc5d4\" (UniqueName: \"kubernetes.io/projected/a848f387-f33e-4c4d-be5b-30757ea54810-kube-api-access-vc5d4\") pod \"test-curl-9092-10c45a\" (UID: \"a848f387-f33e-4c4d-be5b-30757ea54810\") " pod="opendatahub/test-curl-9092-10c45a"
Oct 29 22:36:26.093706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:26.093675    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc5d4\" (UniqueName: \"kubernetes.io/projected/a848f387-f33e-4c4d-be5b-30757ea54810-kube-api-access-vc5d4\") pod \"test-curl-9092-10c45a\" (UID: \"a848f387-f33e-4c4d-be5b-30757ea54810\") " pod="opendatahub/test-curl-9092-10c45a"
Oct 29 22:36:26.271394 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:26.271364    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-10c45a"
Oct 29 22:36:26.734518 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:26.734483    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9092-10c45a"]
Oct 29 22:36:26.923504 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:26.923463    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9092-10c45a" event={"ID":"a848f387-f33e-4c4d-be5b-30757ea54810","Type":"ContainerStarted","Data":"3c47975fb363174d177d93338e88016eeac31b90adaa2657ec132896284ab424"}
Oct 29 22:36:26.923718 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:26.923527    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9092-10c45a" event={"ID":"a848f387-f33e-4c4d-be5b-30757ea54810","Type":"ContainerStarted","Data":"6a05e12429547f52ca0f5dad31f11ccbae3e1213b4ae1f3d6f1aa4a6dc3ca15b"}
Oct 29 22:36:26.923718 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:26.923616    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9092-09835a" podUID="c9899b51-add6-4929-b236-5bd6a1af5257" containerName="test-curl-9092-09835a" containerID="cri-o://d555300eeebc0396753bc2ba4c1c67db5d34eb332d1fb8b430ffc987c17f6879" gracePeriod=2
Oct 29 22:36:26.937915 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:26.937871    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-09835a\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="c9899b51-add6-4929-b236-5bd6a1af5257" pod="opendatahub/test-curl-9092-09835a"
Oct 29 22:36:26.975455 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:26.975398    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9092-10c45a" podStartSLOduration=1.9753812210000001 podStartE2EDuration="1.975381221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:26.974603345 +0000 UTC m=+1325.886046899" watchObservedRunningTime="2026-10-29 22:36:26.975381221 +0000 UTC m=+1325.886824775"
Oct 29 22:36:26.988598 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:26.988504    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-09835a\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="c9899b51-add6-4929-b236-5bd6a1af5257" pod="opendatahub/test-curl-9092-09835a"
Oct 29 22:36:27.838653 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:27.837590    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9092-10c45a"]
Oct 29 22:36:27.841957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:27.841928    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9092-10c45a"]
Oct 29 22:36:27.933022 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:27.932991    4559 generic.go:356] "Generic (PLEG): container finished" podID="aa557804-9bd1-4a83-a858-1d150a4ac0fe" containerID="433dae1e18022d12eed9c784849ede4540a8087123ec41a216b1089dc8d76f7d" exitCode=0
Oct 29 22:36:27.933190 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:27.933061    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh" event={"ID":"aa557804-9bd1-4a83-a858-1d150a4ac0fe","Type":"ContainerDied","Data":"433dae1e18022d12eed9c784849ede4540a8087123ec41a216b1089dc8d76f7d"}
Oct 29 22:36:27.933564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:27.933546    4559 scope.go:122] "RemoveContainer" containerID="433dae1e18022d12eed9c784849ede4540a8087123ec41a216b1089dc8d76f7d"
Oct 29 22:36:27.935090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:27.935065    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:36:27.935158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:27.935133    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:36:27.935194 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:27.935184    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:36:28.006410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:28.006380    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9092-ed962b"]
Oct 29 22:36:28.007420 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:28.007401    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a848f387-f33e-4c4d-be5b-30757ea54810" containerName="test-curl-9092-10c45a"
Oct 29 22:36:28.007507 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:28.007422    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a848f387-f33e-4c4d-be5b-30757ea54810" containerName="test-curl-9092-10c45a"
Oct 29 22:36:28.007586 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:28.007576    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a848f387-f33e-4c4d-be5b-30757ea54810" containerName="test-curl-9092-10c45a"
Oct 29 22:36:28.011290 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:28.011273    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-ed962b"
Oct 29 22:36:28.044796 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:28.044765    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9092-ed962b"]
Oct 29 22:36:28.054712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:28.054587    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5z75\" (UniqueName: \"kubernetes.io/projected/ee5bd003-2421-40be-97ef-f7104d4e09ca-kube-api-access-w5z75\") pod \"test-curl-9092-ed962b\" (UID: \"ee5bd003-2421-40be-97ef-f7104d4e09ca\") " pod="opendatahub/test-curl-9092-ed962b"
Oct 29 22:36:28.159978 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:28.156645    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5z75\" (UniqueName: \"kubernetes.io/projected/ee5bd003-2421-40be-97ef-f7104d4e09ca-kube-api-access-w5z75\") pod \"test-curl-9092-ed962b\" (UID: \"ee5bd003-2421-40be-97ef-f7104d4e09ca\") " pod="opendatahub/test-curl-9092-ed962b"
Oct 29 22:36:28.183611 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:28.183366    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5z75\" (UniqueName: \"kubernetes.io/projected/ee5bd003-2421-40be-97ef-f7104d4e09ca-kube-api-access-w5z75\") pod \"test-curl-9092-ed962b\" (UID: \"ee5bd003-2421-40be-97ef-f7104d4e09ca\") " pod="opendatahub/test-curl-9092-ed962b"
Oct 29 22:36:28.340850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:28.340141    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-ed962b"
Oct 29 22:36:28.680844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:28.680794    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:36:28.770683 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:36:28.770650    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee5bd003_2421_40be_97ef_f7104d4e09ca.slice/crio-eadd3ef7e0125f13ef25de9dd56fe82ebfcb5feaef24c3619cdebe02f691f200 WatchSource:0}: Error finding container eadd3ef7e0125f13ef25de9dd56fe82ebfcb5feaef24c3619cdebe02f691f200: Status 404 returned error can't find the container with id eadd3ef7e0125f13ef25de9dd56fe82ebfcb5feaef24c3619cdebe02f691f200
Oct 29 22:36:28.772120 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:28.772094    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9092-ed962b"]
Oct 29 22:36:28.968523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:28.968487    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh" event={"ID":"aa557804-9bd1-4a83-a858-1d150a4ac0fe","Type":"ContainerStarted","Data":"c7732222d5d1dc5bbda751d8cae745443ec9f2cdb7341b2b716e997e9df1763f"}
Oct 29 22:36:28.970401 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:28.970368    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9092-ed962b" event={"ID":"ee5bd003-2421-40be-97ef-f7104d4e09ca","Type":"ContainerStarted","Data":"5e9ff3d46768cce191b744e6fe21293b0b6d8332a52cddc4e60d36e72a438120"}
Oct 29 22:36:28.970579 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:28.970417    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9092-ed962b" event={"ID":"ee5bd003-2421-40be-97ef-f7104d4e09ca","Type":"ContainerStarted","Data":"eadd3ef7e0125f13ef25de9dd56fe82ebfcb5feaef24c3619cdebe02f691f200"}
Oct 29 22:36:28.970579 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:28.970441    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9092-10c45a" podUID="a848f387-f33e-4c4d-be5b-30757ea54810" containerName="test-curl-9092-10c45a" containerID="cri-o://3c47975fb363174d177d93338e88016eeac31b90adaa2657ec132896284ab424" gracePeriod=2
Oct 29 22:36:29.024961 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:29.024854    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-10c45a\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="a848f387-f33e-4c4d-be5b-30757ea54810" pod="opendatahub/test-curl-9092-10c45a"
Oct 29 22:36:29.226218 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:29.226190    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-09835a"
Oct 29 22:36:29.251551 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:29.242080    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-10c45a\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="a848f387-f33e-4c4d-be5b-30757ea54810" pod="opendatahub/test-curl-9092-10c45a"
Oct 29 22:36:29.301374 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:29.301270    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9899b51-add6-4929-b236-5bd6a1af5257-kube-api-access-w499g\" (UniqueName: \"kubernetes.io/projected/c9899b51-add6-4929-b236-5bd6a1af5257-kube-api-access-w499g\") pod \"c9899b51-add6-4929-b236-5bd6a1af5257\" (UID: \"c9899b51-add6-4929-b236-5bd6a1af5257\") "
Oct 29 22:36:29.303591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:29.303556    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9899b51-add6-4929-b236-5bd6a1af5257-kube-api-access-w499g" pod "c9899b51-add6-4929-b236-5bd6a1af5257" (UID: "c9899b51-add6-4929-b236-5bd6a1af5257"). InnerVolumeSpecName "kube-api-access-w499g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:36:29.403445 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:29.403413    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w499g\" (UniqueName: \"kubernetes.io/projected/c9899b51-add6-4929-b236-5bd6a1af5257-kube-api-access-w499g\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:36:29.910833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:29.897489    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9092-ed962b" podStartSLOduration=2.897469909 podStartE2EDuration="2.897469909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:29.075295957 +0000 UTC m=+1327.986739523" watchObservedRunningTime="2026-10-29 22:36:29.897469909 +0000 UTC m=+1328.808913490"
Oct 29 22:36:29.921836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:29.916390    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:36:29.959831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:29.954292    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:36:29.959831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:29.954371    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh" podUID="aa557804-9bd1-4a83-a858-1d150a4ac0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 22:36:29.984923 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:29.984875    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9092-ed962b"]
Oct 29 22:36:30.097969 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.094550    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9092-ed962b"]
Oct 29 22:36:30.108987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.107639    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9092-154299"]
Oct 29 22:36:30.108987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.108948    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ee5bd003-2421-40be-97ef-f7104d4e09ca" containerName="test-curl-9092-ed962b"
Oct 29 22:36:30.108987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.108973    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5bd003-2421-40be-97ef-f7104d4e09ca" containerName="test-curl-9092-ed962b"
Oct 29 22:36:30.109278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.109132    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5bd003-2421-40be-97ef-f7104d4e09ca" containerName="test-curl-9092-ed962b"
Oct 29 22:36:30.116981 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.116181    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9092-154299"]
Oct 29 22:36:30.116981 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.116285    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-154299"
Oct 29 22:36:30.121237 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.119904    4559 generic.go:356] "Generic (PLEG): container finished" podID="c9899b51-add6-4929-b236-5bd6a1af5257" containerID="d555300eeebc0396753bc2ba4c1c67db5d34eb332d1fb8b430ffc987c17f6879" exitCode=137
Oct 29 22:36:30.121237 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.119985    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-09835a"
Oct 29 22:36:30.121237 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.121097    4559 scope.go:122] "RemoveContainer" containerID="d555300eeebc0396753bc2ba4c1c67db5d34eb332d1fb8b430ffc987c17f6879"
Oct 29 22:36:30.183218 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.183167    4559 scope.go:122] "RemoveContainer" containerID="d555300eeebc0396753bc2ba4c1c67db5d34eb332d1fb8b430ffc987c17f6879"
Oct 29 22:36:30.183562 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:30.183530    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d555300eeebc0396753bc2ba4c1c67db5d34eb332d1fb8b430ffc987c17f6879\": container with ID starting with d555300eeebc0396753bc2ba4c1c67db5d34eb332d1fb8b430ffc987c17f6879 not found: ID does not exist" containerID="d555300eeebc0396753bc2ba4c1c67db5d34eb332d1fb8b430ffc987c17f6879"
Oct 29 22:36:30.183668 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.183573    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d555300eeebc0396753bc2ba4c1c67db5d34eb332d1fb8b430ffc987c17f6879"} err="failed to get container status \"d555300eeebc0396753bc2ba4c1c67db5d34eb332d1fb8b430ffc987c17f6879\": rpc error: code = NotFound desc = could not find container \"d555300eeebc0396753bc2ba4c1c67db5d34eb332d1fb8b430ffc987c17f6879\": container with ID starting with d555300eeebc0396753bc2ba4c1c67db5d34eb332d1fb8b430ffc987c17f6879 not found: ID does not exist"
Oct 29 22:36:30.260458 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.259991    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2tnl\" (UniqueName: \"kubernetes.io/projected/a324066f-11d1-4bd0-856b-8fdf9fe720fb-kube-api-access-k2tnl\") pod \"test-curl-9092-154299\" (UID: \"a324066f-11d1-4bd0-856b-8fdf9fe720fb\") " pod="opendatahub/test-curl-9092-154299"
Oct 29 22:36:30.269794 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:30.269752    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9899b51_add6_4929_b236_5bd6a1af5257.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9899b51_add6_4929_b236_5bd6a1af5257.slice/crio-3414d33dac71599e94542508b94fd30020d49ac87d35ac0beee5c1cfd87a4e33\": RecentStats: unable to find data in memory cache]"
Oct 29 22:36:30.363108 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.362744    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2tnl\" (UniqueName: \"kubernetes.io/projected/a324066f-11d1-4bd0-856b-8fdf9fe720fb-kube-api-access-k2tnl\") pod \"test-curl-9092-154299\" (UID: \"a324066f-11d1-4bd0-856b-8fdf9fe720fb\") " pod="opendatahub/test-curl-9092-154299"
Oct 29 22:36:30.383156 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.383121    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2tnl\" (UniqueName: \"kubernetes.io/projected/a324066f-11d1-4bd0-856b-8fdf9fe720fb-kube-api-access-k2tnl\") pod \"test-curl-9092-154299\" (UID: \"a324066f-11d1-4bd0-856b-8fdf9fe720fb\") " pod="opendatahub/test-curl-9092-154299"
Oct 29 22:36:30.467665 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.467576    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-154299"
Oct 29 22:36:30.793664 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.793622    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9092-154299"]
Oct 29 22:36:30.797037 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:36:30.797007    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda324066f_11d1_4bd0_856b_8fdf9fe720fb.slice/crio-0da428380bde602b5e5a1b12cc5ee3528f2a50d8e5fdf373e0be21078c8ee191 WatchSource:0}: Error finding container 0da428380bde602b5e5a1b12cc5ee3528f2a50d8e5fdf373e0be21078c8ee191: Status 404 returned error can't find the container with id 0da428380bde602b5e5a1b12cc5ee3528f2a50d8e5fdf373e0be21078c8ee191
Oct 29 22:36:30.915166 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.915131    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:36:30.915322 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:30.915210    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh" podUID="aa557804-9bd1-4a83-a858-1d150a4ac0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 22:36:31.132783 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:31.132641    4559 generic.go:356] "Generic (PLEG): container finished" podID="a848f387-f33e-4c4d-be5b-30757ea54810" containerID="3c47975fb363174d177d93338e88016eeac31b90adaa2657ec132896284ab424" exitCode=137
Oct 29 22:36:31.162868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:31.151876    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9092-ed962b" podUID="ee5bd003-2421-40be-97ef-f7104d4e09ca" containerName="test-curl-9092-ed962b" containerID="cri-o://5e9ff3d46768cce191b744e6fe21293b0b6d8332a52cddc4e60d36e72a438120" gracePeriod=2
Oct 29 22:36:31.162868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:31.153144    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9092-154299" event={"ID":"a324066f-11d1-4bd0-856b-8fdf9fe720fb","Type":"ContainerStarted","Data":"2e246910e19038da1b9f024519751a585f4d619ba4300579603211199bfad56a"}
Oct 29 22:36:31.162868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:31.153194    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9092-154299" event={"ID":"a324066f-11d1-4bd0-856b-8fdf9fe720fb","Type":"ContainerStarted","Data":"0da428380bde602b5e5a1b12cc5ee3528f2a50d8e5fdf373e0be21078c8ee191"}
Oct 29 22:36:31.185886 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:31.176414    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-ed962b\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="ee5bd003-2421-40be-97ef-f7104d4e09ca" pod="opendatahub/test-curl-9092-ed962b"
Oct 29 22:36:31.218424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:31.218368    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9092-154299" podStartSLOduration=1.218351846 podStartE2EDuration="1.218351846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:31.214145732 +0000 UTC m=+1330.125589287" watchObservedRunningTime="2026-10-29 22:36:31.218351846 +0000 UTC m=+1330.129795399"
Oct 29 22:36:31.228002 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:31.227961    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-ed962b\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="ee5bd003-2421-40be-97ef-f7104d4e09ca" pod="opendatahub/test-curl-9092-ed962b"
Oct 29 22:36:31.274846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:31.274788    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9899b51-add6-4929-b236-5bd6a1af5257" path="/var/lib/kubelet/pods/c9899b51-add6-4929-b236-5bd6a1af5257/volumes"
Oct 29 22:36:31.284346 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:31.284313    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-ed962b\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="ee5bd003-2421-40be-97ef-f7104d4e09ca" pod="opendatahub/test-curl-9092-ed962b"
Oct 29 22:36:31.292318 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:31.292292    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-10c45a"
Oct 29 22:36:31.313596 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:31.313553    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-10c45a\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="a848f387-f33e-4c4d-be5b-30757ea54810" pod="opendatahub/test-curl-9092-10c45a"
Oct 29 22:36:31.389668 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:31.389565    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a848f387-f33e-4c4d-be5b-30757ea54810-kube-api-access-vc5d4\" (UniqueName: \"kubernetes.io/projected/a848f387-f33e-4c4d-be5b-30757ea54810-kube-api-access-vc5d4\") pod \"a848f387-f33e-4c4d-be5b-30757ea54810\" (UID: \"a848f387-f33e-4c4d-be5b-30757ea54810\") "
Oct 29 22:36:31.392284 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:31.392252    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a848f387-f33e-4c4d-be5b-30757ea54810-kube-api-access-vc5d4" pod "a848f387-f33e-4c4d-be5b-30757ea54810" (UID: "a848f387-f33e-4c4d-be5b-30757ea54810"). InnerVolumeSpecName "kube-api-access-vc5d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:36:31.492641 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:31.492604    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc5d4\" (UniqueName: \"kubernetes.io/projected/a848f387-f33e-4c4d-be5b-30757ea54810-kube-api-access-vc5d4\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:36:31.915852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:31.915788    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:36:31.916029 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:31.915925    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh" podUID="aa557804-9bd1-4a83-a858-1d150a4ac0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 22:36:32.129498 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:32.129458    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9092-154299"]
Oct 29 22:36:32.169900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:32.165129    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9092-154299"]
Oct 29 22:36:32.204652 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:32.202791    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-10c45a"
Oct 29 22:36:32.204652 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:32.202880    4559 scope.go:122] "RemoveContainer" containerID="3c47975fb363174d177d93338e88016eeac31b90adaa2657ec132896284ab424"
Oct 29 22:36:32.219908 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:32.217201    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-10c45a\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="a848f387-f33e-4c4d-be5b-30757ea54810" pod="opendatahub/test-curl-9092-10c45a"
Oct 29 22:36:32.244837 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:32.239067    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-10c45a\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="a848f387-f33e-4c4d-be5b-30757ea54810" pod="opendatahub/test-curl-9092-10c45a"
Oct 29 22:36:32.916140 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:32.916105    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:36:32.916334 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:32.916162    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh" podUID="aa557804-9bd1-4a83-a858-1d150a4ac0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 22:36:33.105287 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.103148    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"]
Oct 29 22:36:33.105287 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.104407    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a324066f-11d1-4bd0-856b-8fdf9fe720fb" containerName="test-curl-9092-154299"
Oct 29 22:36:33.105287 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.104427    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a324066f-11d1-4bd0-856b-8fdf9fe720fb" containerName="test-curl-9092-154299"
Oct 29 22:36:33.105287 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.104609    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a324066f-11d1-4bd0-856b-8fdf9fe720fb" containerName="test-curl-9092-154299"
Oct 29 22:36:33.119597 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.116528    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.120572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.116405    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"]
Oct 29 22:36:33.169955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.157207    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cae55b-openshift-default-dockercfg-pxrq8\""
Oct 29 22:36:33.169955 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:33.157306    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-10c45a\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="a848f387-f33e-4c4d-be5b-30757ea54810" pod="opendatahub/test-curl-9092-10c45a"
Oct 29 22:36:33.217355 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.215324    4559 generic.go:356] "Generic (PLEG): container finished" podID="ee5bd003-2421-40be-97ef-f7104d4e09ca" containerID="5e9ff3d46768cce191b744e6fe21293b0b6d8332a52cddc4e60d36e72a438120" exitCode=137
Oct 29 22:36:33.217355 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.217075    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9092-154299" podUID="a324066f-11d1-4bd0-856b-8fdf9fe720fb" containerName="test-curl-9092-154299" containerID="cri-o://2e246910e19038da1b9f024519751a585f4d619ba4300579603211199bfad56a" gracePeriod=2
Oct 29 22:36:33.231180 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:33.231034    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-154299\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="a324066f-11d1-4bd0-856b-8fdf9fe720fb" pod="opendatahub/test-curl-9092-154299"
Oct 29 22:36:33.254090 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:33.254051    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-10c45a\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="a848f387-f33e-4c4d-be5b-30757ea54810" pod="opendatahub/test-curl-9092-10c45a"
Oct 29 22:36:33.277803 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.277679    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-credential-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.279896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.279863    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-token\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.280154 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.280040    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95km4\" (UniqueName: \"kubernetes.io/projected/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-kube-api-access-95km4\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.280217 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.280183    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-workload-certs\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.280345 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.280328    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-data\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.280501 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.280481    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-podinfo\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.280993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.280971    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-envoy\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.281434 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.281413    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-workload-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.281752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.281731    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istiod-ca-cert\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.287972 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.287940    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a848f387-f33e-4c4d-be5b-30757ea54810" path="/var/lib/kubelet/pods/a848f387-f33e-4c4d-be5b-30757ea54810/volumes"
Oct 29 22:36:33.413835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.411324    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-workload-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.585035 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.584400    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-workload-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.663850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.643091    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istiod-ca-cert\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.663850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.643211    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-credential-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.663850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.643294    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-token\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.663850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.643358    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95km4\" (UniqueName: \"kubernetes.io/projected/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-kube-api-access-95km4\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.663850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.643392    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-workload-certs\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.663850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.643440    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-data\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.663850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.643492    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-podinfo\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.663850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.643722    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-envoy\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.663850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.660695    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-workload-certs\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.735939 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.690754    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-credential-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.735939 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.691741    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-data\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.771835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.743305    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istiod-ca-cert\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.903894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.887938    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-token\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.903894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.895554    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-envoy\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.903894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.898898    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95km4\" (UniqueName: \"kubernetes.io/projected/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-kube-api-access-95km4\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:33.918719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.918236    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:36:33.918719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.918300    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh" podUID="aa557804-9bd1-4a83-a858-1d150a4ac0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 22:36:33.942838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:33.936900    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-podinfo\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:34.009839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:34.005666    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-ed962b"
Oct 29 22:36:34.097406 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:34.097368    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5bd003-2421-40be-97ef-f7104d4e09ca-kube-api-access-w5z75\" (UniqueName: \"kubernetes.io/projected/ee5bd003-2421-40be-97ef-f7104d4e09ca-kube-api-access-w5z75\") pod \"ee5bd003-2421-40be-97ef-f7104d4e09ca\" (UID: \"ee5bd003-2421-40be-97ef-f7104d4e09ca\") "
Oct 29 22:36:34.106289 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:34.106178    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5bd003-2421-40be-97ef-f7104d4e09ca-kube-api-access-w5z75" pod "ee5bd003-2421-40be-97ef-f7104d4e09ca" (UID: "ee5bd003-2421-40be-97ef-f7104d4e09ca"). InnerVolumeSpecName "kube-api-access-w5z75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:36:34.192929 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:34.192368    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:34.260837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:34.250471    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5z75\" (UniqueName: \"kubernetes.io/projected/ee5bd003-2421-40be-97ef-f7104d4e09ca-kube-api-access-w5z75\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:36:34.288884 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:34.288453    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-ed962b"
Oct 29 22:36:34.288884 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:34.288514    4559 scope.go:122] "RemoveContainer" containerID="5e9ff3d46768cce191b744e6fe21293b0b6d8332a52cddc4e60d36e72a438120"
Oct 29 22:36:34.321645 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:34.321606    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-ed962b\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="ee5bd003-2421-40be-97ef-f7104d4e09ca" pod="opendatahub/test-curl-9092-ed962b"
Oct 29 22:36:34.615911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:34.615882    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"]
Oct 29 22:36:34.616058 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:36:34.616008    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5762b8_dbb2_4967_a9c5_8a2876e913fb.slice/crio-3eb0c2d3b71c4659c96383d99b5a2d30c366a3c8009be006f4606adf0f2b9f86 WatchSource:0}: Error finding container 3eb0c2d3b71c4659c96383d99b5a2d30c366a3c8009be006f4606adf0f2b9f86: Status 404 returned error can't find the container with id 3eb0c2d3b71c4659c96383d99b5a2d30c366a3c8009be006f4606adf0f2b9f86
Oct 29 22:36:34.619472 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:34.619431    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:36:34.619596 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:34.619540    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:36:34.619657 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:34.619590    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:36:34.921533 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:34.921503    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:36:34.921867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:34.921845    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:36:34.922642 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:34.922620    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 22:36:34.980152 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:34.980098    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-ed962b\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="ee5bd003-2421-40be-97ef-f7104d4e09ca" pod="opendatahub/test-curl-9092-ed962b"
Oct 29 22:36:34.994166 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:34.994132    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-ed962b\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="ee5bd003-2421-40be-97ef-f7104d4e09ca" pod="opendatahub/test-curl-9092-ed962b"
Oct 29 22:36:35.230383 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:35.230352    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5bd003-2421-40be-97ef-f7104d4e09ca" path="/var/lib/kubelet/pods/ee5bd003-2421-40be-97ef-f7104d4e09ca/volumes"
Oct 29 22:36:35.441617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:35.436877    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" event={"ID":"bf5762b8-dbb2-4967-a9c5-8a2876e913fb","Type":"ContainerStarted","Data":"5fbc65ca6eaab59555eb417db20e1b9251264228558ca765d8a14f9f167efae2"}
Oct 29 22:36:35.441617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:35.436960    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" event={"ID":"bf5762b8-dbb2-4967-a9c5-8a2876e913fb","Type":"ContainerStarted","Data":"3eb0c2d3b71c4659c96383d99b5a2d30c366a3c8009be006f4606adf0f2b9f86"}
Oct 29 22:36:35.441617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:35.440425    4559 generic.go:356] "Generic (PLEG): container finished" podID="a324066f-11d1-4bd0-856b-8fdf9fe720fb" containerID="2e246910e19038da1b9f024519751a585f4d619ba4300579603211199bfad56a" exitCode=137
Oct 29 22:36:35.485943 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:35.485835    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" podStartSLOduration=2.485797594 podStartE2EDuration="2.485797594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:35.481155603 +0000 UTC m=+1334.392599158" watchObservedRunningTime="2026-10-29 22:36:35.485797594 +0000 UTC m=+1334.397241149"
Oct 29 22:36:35.739249 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:35.733916    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-154299"
Oct 29 22:36:35.820016 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:35.819979    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a324066f-11d1-4bd0-856b-8fdf9fe720fb-kube-api-access-k2tnl\" (UniqueName: \"kubernetes.io/projected/a324066f-11d1-4bd0-856b-8fdf9fe720fb-kube-api-access-k2tnl\") pod \"a324066f-11d1-4bd0-856b-8fdf9fe720fb\" (UID: \"a324066f-11d1-4bd0-856b-8fdf9fe720fb\") "
Oct 29 22:36:35.822555 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:35.822526    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a324066f-11d1-4bd0-856b-8fdf9fe720fb-kube-api-access-k2tnl" pod "a324066f-11d1-4bd0-856b-8fdf9fe720fb" (UID: "a324066f-11d1-4bd0-856b-8fdf9fe720fb"). InnerVolumeSpecName "kube-api-access-k2tnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:36:35.922742 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:35.922698    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2tnl\" (UniqueName: \"kubernetes.io/projected/a324066f-11d1-4bd0-856b-8fdf9fe720fb-kube-api-access-k2tnl\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:36:36.192997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:36.192959    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:36.199560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:36.199530    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:36.452159 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:36.452067    4559 scope.go:122] "RemoveContainer" containerID="2e246910e19038da1b9f024519751a585f4d619ba4300579603211199bfad56a"
Oct 29 22:36:36.452159 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:36.452094    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9092-154299"
Oct 29 22:36:36.452688 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:36.452652    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:36.453752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:36.453730    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:36:36.508525 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:36.508480    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9092-154299\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="a324066f-11d1-4bd0-856b-8fdf9fe720fb" pod="opendatahub/test-curl-9092-154299"
Oct 29 22:36:37.229514 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:37.229483    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a324066f-11d1-4bd0-856b-8fdf9fe720fb" path="/var/lib/kubelet/pods/a324066f-11d1-4bd0-856b-8fdf9fe720fb/volumes"
Oct 29 22:36:38.198235 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.198191    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"]
Oct 29 22:36:38.205684 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.205654    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.237327 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.237266    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1fa4b8b6\"/\"openshift-service-ca.crt\""
Oct 29 22:36:38.237327 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.237289    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1fa4b8b6\"/\"default-dockercfg-nlps7\""
Oct 29 22:36:38.237569 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.237343    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1fa4b8b6\"/\"e2e-lifecycle-model-1fa4b8b6-kserve-self-signed-certs\""
Oct 29 22:36:38.237569 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.237415    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1fa4b8b6\"/\"kube-root-ca.crt\""
Oct 29 22:36:38.244699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.244669    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"]
Oct 29 22:36:38.273010 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.272980    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-home\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.273185 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.273054    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-tls-certs\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.273185 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.273166    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-dshm\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.273340 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.273322    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-tmp-dir\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.273457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.273442    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-model-cache\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.273493 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.273478    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqb6z\" (UniqueName: \"kubernetes.io/projected/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-kube-api-access-bqb6z\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.374862 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.374823    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-tls-certs\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.375056 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.374895    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-dshm\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.375056 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.375035    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-tmp-dir\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.375190 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.375166    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-model-cache\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.375248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.375219    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqb6z\" (UniqueName: \"kubernetes.io/projected/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-kube-api-access-bqb6z\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.375423 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.375403    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-home\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.375584 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.375554    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-tmp-dir\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.375872 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.375838    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-model-cache\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.376128 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.375801    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-home\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.377693 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.377667    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-dshm\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.378190 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.378169    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-tls-certs\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.402155 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.402117    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqb6z\" (UniqueName: \"kubernetes.io/projected/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-kube-api-access-bqb6z\") pod \"e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") " pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:38.523886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:38.523554    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:39.089519 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:36:39.089444    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf31c5a3f_ff14_4fb7_b42c_ed9c9d2c2a73.slice/crio-23700c9e37793af0793db7a1fc016c4c460483fe6059d7facb5144c5aea0edef WatchSource:0}: Error finding container 23700c9e37793af0793db7a1fc016c4c460483fe6059d7facb5144c5aea0edef: Status 404 returned error can't find the container with id 23700c9e37793af0793db7a1fc016c4c460483fe6059d7facb5144c5aea0edef
Oct 29 22:36:39.090873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.090747    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"]
Oct 29 22:36:39.497304 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.497262    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" event={"ID":"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73","Type":"ContainerStarted","Data":"8b0d6b0b531785984d32f9bad16c126d31c415e5148631ea31bd92023390a36a"}
Oct 29 22:36:39.497754 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.497323    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" event={"ID":"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73","Type":"ContainerStarted","Data":"23700c9e37793af0793db7a1fc016c4c460483fe6059d7facb5144c5aea0edef"}
Oct 29 22:36:39.524553 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.524521    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:39.542959 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.542901    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" podStartSLOduration=1.542886154 podStartE2EDuration="1.542886154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:39.540018712 +0000 UTC m=+1338.451462269" watchObservedRunningTime="2026-10-29 22:36:39.542886154 +0000 UTC m=+1338.454329707"
Oct 29 22:36:39.924789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.924753    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"]
Oct 29 22:36:39.930543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.930520    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:39.971231 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.971195    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-616da7-openshift-default-dockercfg-s2jwm\""
Oct 29 22:36:39.978985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.978946    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"]
Oct 29 22:36:39.987299 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.987260    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-token\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:39.987469 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.987338    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-workload-certs\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:39.987469 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.987365    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-envoy\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:39.987609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.987500    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j24c4\" (UniqueName: \"kubernetes.io/projected/2040bbf0-5596-401b-b7f6-b3fa571c4d94-kube-api-access-j24c4\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:39.987609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.987557    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-credential-socket\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:39.987609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.987604    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istiod-ca-cert\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:39.987748 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.987662    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-data\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:39.988018 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.987853    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-workload-socket\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:39.988098 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:39.988060    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-podinfo\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.166852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.123421    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-workload-socket\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.166852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.123468    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-podinfo\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.166852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.123650    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-token\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.166852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.123702    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-workload-certs\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.166852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.123726    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-envoy\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.166852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.123874    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j24c4\" (UniqueName: \"kubernetes.io/projected/2040bbf0-5596-401b-b7f6-b3fa571c4d94-kube-api-access-j24c4\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.166852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.123919    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-credential-socket\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.166852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.123962    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istiod-ca-cert\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.166852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.124015    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-data\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.166852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.124543    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-data\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.166852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.125102    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-workload-certs\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.166852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.137506    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-credential-socket\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.166852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.138447    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istiod-ca-cert\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.364668 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.364561    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-envoy\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.372938 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.365327    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-workload-socket\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.391492 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.391424    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-podinfo\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.575854 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.566884    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j24c4\" (UniqueName: \"kubernetes.io/projected/2040bbf0-5596-401b-b7f6-b3fa571c4d94-kube-api-access-j24c4\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.582455 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.582219    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"]
Oct 29 22:36:40.606833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.603415    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-token\") pod \"e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") " pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.642960 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.636534    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"]
Oct 29 22:36:40.642960 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.636592    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"]
Oct 29 22:36:40.647891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.647863    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"
Oct 29 22:36:40.651445 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.651418    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"]
Oct 29 22:36:40.651758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.651741    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"
Oct 29 22:36:40.680194 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.680153    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cae55b\""
Oct 29 22:36:40.680682 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.680659    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cae55b-dockercfg-kr9w4\""
Oct 29 22:36:40.685719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.685672    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl4mz\" (UniqueName: \"kubernetes.io/projected/8f5cb439-a622-48d5-9b98-b76c8b94dd85-kube-api-access-zl4mz\") pod \"payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2\" (UID: \"8f5cb439-a622-48d5-9b98-b76c8b94dd85\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"
Oct 29 22:36:40.686051 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.686023    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f5cb439-a622-48d5-9b98-b76c8b94dd85-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2\" (UID: \"8f5cb439-a622-48d5-9b98-b76c8b94dd85\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"
Oct 29 22:36:40.686509 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.686477    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rnd8\" (UniqueName: \"kubernetes.io/projected/7cb532d0-c0f2-4a78-94ce-5b6272b175e5-kube-api-access-9rnd8\") pod \"payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59\" (UID: \"7cb532d0-c0f2-4a78-94ce-5b6272b175e5\") " pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"
Oct 29 22:36:40.686746 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.686731    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb532d0-c0f2-4a78-94ce-5b6272b175e5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59\" (UID: \"7cb532d0-c0f2-4a78-94ce-5b6272b175e5\") " pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"
Oct 29 22:36:40.825989 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.825940    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb532d0-c0f2-4a78-94ce-5b6272b175e5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59\" (UID: \"7cb532d0-c0f2-4a78-94ce-5b6272b175e5\") " pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"
Oct 29 22:36:40.826319 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.826287    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl4mz\" (UniqueName: \"kubernetes.io/projected/8f5cb439-a622-48d5-9b98-b76c8b94dd85-kube-api-access-zl4mz\") pod \"payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2\" (UID: \"8f5cb439-a622-48d5-9b98-b76c8b94dd85\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"
Oct 29 22:36:40.826588 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.826568    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f5cb439-a622-48d5-9b98-b76c8b94dd85-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2\" (UID: \"8f5cb439-a622-48d5-9b98-b76c8b94dd85\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"
Oct 29 22:36:40.827477 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.827453    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rnd8\" (UniqueName: \"kubernetes.io/projected/7cb532d0-c0f2-4a78-94ce-5b6272b175e5-kube-api-access-9rnd8\") pod \"payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59\" (UID: \"7cb532d0-c0f2-4a78-94ce-5b6272b175e5\") " pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"
Oct 29 22:36:40.829218 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.829183    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb532d0-c0f2-4a78-94ce-5b6272b175e5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59\" (UID: \"7cb532d0-c0f2-4a78-94ce-5b6272b175e5\") " pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"
Oct 29 22:36:40.841879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.835954    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f5cb439-a622-48d5-9b98-b76c8b94dd85-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2\" (UID: \"8f5cb439-a622-48d5-9b98-b76c8b94dd85\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"
Oct 29 22:36:40.861259 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.861216    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:40.910336 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.909383    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl4mz\" (UniqueName: \"kubernetes.io/projected/8f5cb439-a622-48d5-9b98-b76c8b94dd85-kube-api-access-zl4mz\") pod \"payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2\" (UID: \"8f5cb439-a622-48d5-9b98-b76c8b94dd85\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"
Oct 29 22:36:40.910336 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:40.909654    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rnd8\" (UniqueName: \"kubernetes.io/projected/7cb532d0-c0f2-4a78-94ce-5b6272b175e5-kube-api-access-9rnd8\") pod \"payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59\" (UID: \"7cb532d0-c0f2-4a78-94ce-5b6272b175e5\") " pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"
Oct 29 22:36:41.157406 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:41.157128    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"
Oct 29 22:36:41.229395 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:41.179185    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"
Oct 29 22:36:41.361322 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:41.361269    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"]
Oct 29 22:36:41.484042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:41.483740    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:36:41.486060 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:41.486021    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:36:41.502776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:41.502721    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:36:41.646330 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:41.645746    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" event={"ID":"2040bbf0-5596-401b-b7f6-b3fa571c4d94","Type":"ContainerStarted","Data":"2ba7f3492cf1c8b567d55d228117523fbe0a6be6f8226c06cbfce4b44af214e4"}
Oct 29 22:36:41.836911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:41.836877    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"]
Oct 29 22:36:41.882543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:41.882505    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"]
Oct 29 22:36:41.888943 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:36:41.888914    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb532d0_c0f2_4a78_94ce_5b6272b175e5.slice/crio-aa3ca0a9c23c1f80f44f01d29b4a17a12dd72d1b744cd9d17ba7f3518dda97e4 WatchSource:0}: Error finding container aa3ca0a9c23c1f80f44f01d29b4a17a12dd72d1b744cd9d17ba7f3518dda97e4: Status 404 returned error can't find the container with id aa3ca0a9c23c1f80f44f01d29b4a17a12dd72d1b744cd9d17ba7f3518dda97e4
Oct 29 22:36:42.706347 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:42.701337    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" event={"ID":"2040bbf0-5596-401b-b7f6-b3fa571c4d94","Type":"ContainerStarted","Data":"fe86c21f19f80948a127927d9d77da3248c1bd765ff7f602c4c0bd0b01f22b92"}
Oct 29 22:36:42.715123 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:42.715082    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59" event={"ID":"7cb532d0-c0f2-4a78-94ce-5b6272b175e5","Type":"ContainerStarted","Data":"cf43416fec668be01b6c184a76063f61b0fcce335935d4372d52fdb85d45481b"}
Oct 29 22:36:42.715288 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:42.715151    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59" event={"ID":"7cb532d0-c0f2-4a78-94ce-5b6272b175e5","Type":"ContainerStarted","Data":"aa3ca0a9c23c1f80f44f01d29b4a17a12dd72d1b744cd9d17ba7f3518dda97e4"}
Oct 29 22:36:42.715288 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:42.715276    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"
Oct 29 22:36:42.716902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:42.716870    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2" event={"ID":"8f5cb439-a622-48d5-9b98-b76c8b94dd85","Type":"ContainerStarted","Data":"3476f7edcf2c0be6eb10b7aa1370bcd284b33c96724e227e0c17495f6c7496d2"}
Oct 29 22:36:42.716984 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:42.716931    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2" event={"ID":"8f5cb439-a622-48d5-9b98-b76c8b94dd85","Type":"ContainerStarted","Data":"9ee242ba187652d94f46d64f55c6933dbba18b0ca3b3e69a8094e5f86b0cdfe6"}
Oct 29 22:36:42.716984 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:42.716962    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"
Oct 29 22:36:42.763332 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:42.763261    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" podStartSLOduration=3.763240063 podStartE2EDuration="3.763240063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:42.759095711 +0000 UTC m=+1341.670539265" watchObservedRunningTime="2026-10-29 22:36:42.763240063 +0000 UTC m=+1341.674683621"
Oct 29 22:36:42.800572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:42.800512    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2" podStartSLOduration=2.800496779 podStartE2EDuration="2.800496779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:42.797117224 +0000 UTC m=+1341.708560777" watchObservedRunningTime="2026-10-29 22:36:42.800496779 +0000 UTC m=+1341.711940333"
Oct 29 22:36:42.844063 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:42.844012    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59" podStartSLOduration=2.8439983939999998 podStartE2EDuration="2.843998394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:42.840555106 +0000 UTC m=+1341.751998661" watchObservedRunningTime="2026-10-29 22:36:42.843998394 +0000 UTC m=+1341.755441948"
Oct 29 22:36:42.862823 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:42.862776    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:42.870887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:42.870856    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:43.695278 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:36:43.695246    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:36:43.726027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:43.725994    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:43.727034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:43.727010    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:36:45.319349 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:45.319313    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:36:47.302542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:47.302504    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"]
Oct 29 22:36:47.310573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:47.310545    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"
Oct 29 22:36:47.333487 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:47.333448    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-616da7\""
Oct 29 22:36:47.341460 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:47.341399    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"]
Oct 29 22:36:47.440837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:47.398696    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-616da7-dockercfg-7p7n9\""
Oct 29 22:36:47.454094 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:47.454050    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4nnd\" (UniqueName: \"kubernetes.io/projected/89e9567e-9768-4800-8fee-cbe5b93d7850-kube-api-access-f4nnd\") pod \"payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj\" (UID: \"89e9567e-9768-4800-8fee-cbe5b93d7850\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"
Oct 29 22:36:47.595099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:47.595004    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e9567e-9768-4800-8fee-cbe5b93d7850-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj\" (UID: \"89e9567e-9768-4800-8fee-cbe5b93d7850\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"
Oct 29 22:36:47.645636 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:47.610113    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"]
Oct 29 22:36:47.727633 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:47.727587    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e9567e-9768-4800-8fee-cbe5b93d7850-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj\" (UID: \"89e9567e-9768-4800-8fee-cbe5b93d7850\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"
Oct 29 22:36:47.743251 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:47.729723    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e9567e-9768-4800-8fee-cbe5b93d7850-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj\" (UID: \"89e9567e-9768-4800-8fee-cbe5b93d7850\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"
Oct 29 22:36:47.746059 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:47.746021    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4nnd\" (UniqueName: \"kubernetes.io/projected/89e9567e-9768-4800-8fee-cbe5b93d7850-kube-api-access-f4nnd\") pod \"payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj\" (UID: \"89e9567e-9768-4800-8fee-cbe5b93d7850\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"
Oct 29 22:36:47.787721 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:47.787685    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4nnd\" (UniqueName: \"kubernetes.io/projected/89e9567e-9768-4800-8fee-cbe5b93d7850-kube-api-access-f4nnd\") pod \"payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj\" (UID: \"89e9567e-9768-4800-8fee-cbe5b93d7850\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"
Oct 29 22:36:47.855941 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:47.850123    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"
Oct 29 22:36:47.983119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:47.983091    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"]
Oct 29 22:36:47.991934 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:47.984801    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a99d09-6876-4624-99f0-3fb56f3b3935-plugins-config-volume\") pod \"payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6\" (UID: \"63a99d09-6876-4624-99f0-3fb56f3b3935\") " pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"
Oct 29 22:36:47.991934 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:47.987786    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb2cc\" (UniqueName: \"kubernetes.io/projected/63a99d09-6876-4624-99f0-3fb56f3b3935-kube-api-access-rb2cc\") pod \"payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6\" (UID: \"63a99d09-6876-4624-99f0-3fb56f3b3935\") " pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"
Oct 29 22:36:48.018839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:48.001413    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"
Oct 29 22:36:48.123941 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:48.123093    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb2cc\" (UniqueName: \"kubernetes.io/projected/63a99d09-6876-4624-99f0-3fb56f3b3935-kube-api-access-rb2cc\") pod \"payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6\" (UID: \"63a99d09-6876-4624-99f0-3fb56f3b3935\") " pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"
Oct 29 22:36:48.123941 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:48.123565    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a99d09-6876-4624-99f0-3fb56f3b3935-plugins-config-volume\") pod \"payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6\" (UID: \"63a99d09-6876-4624-99f0-3fb56f3b3935\") " pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"
Oct 29 22:36:48.127290 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:48.125119    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a99d09-6876-4624-99f0-3fb56f3b3935-plugins-config-volume\") pod \"payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6\" (UID: \"63a99d09-6876-4624-99f0-3fb56f3b3935\") " pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"
Oct 29 22:36:48.192963 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:48.159719    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb2cc\" (UniqueName: \"kubernetes.io/projected/63a99d09-6876-4624-99f0-3fb56f3b3935-kube-api-access-rb2cc\") pod \"payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6\" (UID: \"63a99d09-6876-4624-99f0-3fb56f3b3935\") " pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"
Oct 29 22:36:48.307936 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:48.307242    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"
Oct 29 22:36:48.561765 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:48.560688    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:48.584772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:48.584744    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:48.677976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:48.677939    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"]
Oct 29 22:36:48.980647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:48.978356    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"]
Oct 29 22:36:48.992955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:48.992920    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj" event={"ID":"89e9567e-9768-4800-8fee-cbe5b93d7850","Type":"ContainerStarted","Data":"6dbefe001dca9fd07ddd36e9216c8339ff22890dc8e154a53115f57b058ad0e9"}
Oct 29 22:36:48.993176 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:48.993151    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj" event={"ID":"89e9567e-9768-4800-8fee-cbe5b93d7850","Type":"ContainerStarted","Data":"972bb4093f788fe5e85be6c1891aff272fbbe301419550d6ae0d6592bdfa88e6"}
Oct 29 22:36:48.993346 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:48.993328    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"
Oct 29 22:36:49.028969 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:49.028935    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:36:49.066303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:49.066235    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj" podStartSLOduration=2.066213775 podStartE2EDuration="2.066213775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:49.032151292 +0000 UTC m=+1347.943594872" watchObservedRunningTime="2026-10-29 22:36:49.066213775 +0000 UTC m=+1347.977657330"
Oct 29 22:36:50.225871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:50.201709    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6" event={"ID":"63a99d09-6876-4624-99f0-3fb56f3b3935","Type":"ContainerStarted","Data":"057f7a5667864e004ebe1c0e2867ef8c7845e90a34340aff87fa03f961f94fef"}
Oct 29 22:36:50.225871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:50.224005    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"
Oct 29 22:36:50.225871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:50.224098    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6" event={"ID":"63a99d09-6876-4624-99f0-3fb56f3b3935","Type":"ContainerStarted","Data":"5192e775069da513e90a847626d3b37d9c17acaf4dec86153c1a617af6a5a1e1"}
Oct 29 22:36:50.260524 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:50.257504    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6" podStartSLOduration=3.2574870430000002 podStartE2EDuration="3.257487043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:50.256865485 +0000 UTC m=+1349.168309052" watchObservedRunningTime="2026-10-29 22:36:50.257487043 +0000 UTC m=+1349.168930598"
Oct 29 22:36:52.315837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:52.304914    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"]
Oct 29 22:36:52.642900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:52.641506    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"
Oct 29 22:36:52.666989 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:52.650843    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"]
Oct 29 22:36:52.666989 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:52.656786    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ee886f\""
Oct 29 22:36:52.688878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:52.670128    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ee886f-dockercfg-dcs78\""
Oct 29 22:36:52.759262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:52.745128    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd88602-acf5-41be-89f4-e5aa49e2a357-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn\" (UID: \"2dd88602-acf5-41be-89f4-e5aa49e2a357\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"
Oct 29 22:36:52.759262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:52.745443    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvcl8\" (UniqueName: \"kubernetes.io/projected/2dd88602-acf5-41be-89f4-e5aa49e2a357-kube-api-access-jvcl8\") pod \"payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn\" (UID: \"2dd88602-acf5-41be-89f4-e5aa49e2a357\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"
Oct 29 22:36:52.847893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:52.847856    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd88602-acf5-41be-89f4-e5aa49e2a357-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn\" (UID: \"2dd88602-acf5-41be-89f4-e5aa49e2a357\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"
Oct 29 22:36:52.848553 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:52.848522    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvcl8\" (UniqueName: \"kubernetes.io/projected/2dd88602-acf5-41be-89f4-e5aa49e2a357-kube-api-access-jvcl8\") pod \"payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn\" (UID: \"2dd88602-acf5-41be-89f4-e5aa49e2a357\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"
Oct 29 22:36:52.857370 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:52.857339    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd88602-acf5-41be-89f4-e5aa49e2a357-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn\" (UID: \"2dd88602-acf5-41be-89f4-e5aa49e2a357\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"
Oct 29 22:36:52.903303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:52.903182    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvcl8\" (UniqueName: \"kubernetes.io/projected/2dd88602-acf5-41be-89f4-e5aa49e2a357-kube-api-access-jvcl8\") pod \"payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn\" (UID: \"2dd88602-acf5-41be-89f4-e5aa49e2a357\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"
Oct 29 22:36:53.098679 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:53.098642    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"
Oct 29 22:36:53.416488 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:36:53.416456    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd88602_acf5_41be_89f4_e5aa49e2a357.slice/crio-5ec11230d189450edaf1b37f85788502fb53ae241c918af245b078ee5e827d69 WatchSource:0}: Error finding container 5ec11230d189450edaf1b37f85788502fb53ae241c918af245b078ee5e827d69: Status 404 returned error can't find the container with id 5ec11230d189450edaf1b37f85788502fb53ae241c918af245b078ee5e827d69
Oct 29 22:36:53.418379 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:53.418346    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"]
Oct 29 22:36:53.653782 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:53.653748    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn" event={"ID":"2dd88602-acf5-41be-89f4-e5aa49e2a357","Type":"ContainerStarted","Data":"3c229ca553c1469080f7d278e9951ba06002e1f778b7c272ed657a45474971f5"}
Oct 29 22:36:53.653985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:53.653823    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn" event={"ID":"2dd88602-acf5-41be-89f4-e5aa49e2a357","Type":"ContainerStarted","Data":"5ec11230d189450edaf1b37f85788502fb53ae241c918af245b078ee5e827d69"}
Oct 29 22:36:53.654045 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:53.654000    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"
Oct 29 22:36:53.695162 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:53.695043    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn" podStartSLOduration=1.6950237540000002 podStartE2EDuration="1.695023754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:53.693927146 +0000 UTC m=+1352.605370700" watchObservedRunningTime="2026-10-29 22:36:53.695023754 +0000 UTC m=+1352.606467309"
Oct 29 22:36:53.728685 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:53.728652    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"
Oct 29 22:36:53.728883 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:36:53.728723    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"
Oct 29 22:37:00.204140 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:00.204098    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"
Oct 29 22:37:01.327570 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:01.327541    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"
Oct 29 22:37:02.033826 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:02.033769    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"]
Oct 29 22:37:02.048656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:02.048617    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c95b37-vlgjr"]
Oct 29 22:37:03.231957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:03.231923    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53fdfe33-36d4-493a-ac2b-520cdd5977fa" path="/var/lib/kubelet/pods/53fdfe33-36d4-493a-ac2b-520cdd5977fa/volumes"
Oct 29 22:37:04.063826 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:04.063768    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"]
Oct 29 22:37:04.097774 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:04.097735    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1696adc8-5vf6w"]
Oct 29 22:37:04.672638 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:04.672609    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"
Oct 29 22:37:05.231214 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:05.231167    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d23143c1-8b6a-4975-a219-832af8f64116" path="/var/lib/kubelet/pods/d23143c1-8b6a-4975-a219-832af8f64116/volumes"
Oct 29 22:37:13.697514 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:37:13.697474    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:37:14.016242 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.016157    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"]
Oct 29 22:37:14.021313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.021288    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.132850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.114838    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.143125 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.143082    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8682c2-1888-4569-9b49-a12f036b968b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.143321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.143293    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8682c2-1888-4569-9b49-a12f036b968b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.143512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.143489    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.143611 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.143591    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.143668 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.143639    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.143722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.143702    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8682c2-1888-4569-9b49-a12f036b968b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.143852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.143834    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.210528 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.177273    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"]
Oct 29 22:37:14.262517 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.262457    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.262863 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.262838    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.263044 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.263028    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.263248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.263228    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8682c2-1888-4569-9b49-a12f036b968b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.263457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.263442    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.263634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.263617    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4qzf\" (UniqueName: \"kubernetes.io/projected/1a8682c2-1888-4569-9b49-a12f036b968b-kube-api-access-p4qzf\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.264099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.264078    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.264410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.264391    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8682c2-1888-4569-9b49-a12f036b968b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.265743 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.265718    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8682c2-1888-4569-9b49-a12f036b968b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.267103 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.267044    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.267307 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.267287    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.267451 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.267429    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.267861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.267842    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.268162 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.268138    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8682c2-1888-4569-9b49-a12f036b968b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.270310 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.270289    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8682c2-1888-4569-9b49-a12f036b968b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.273279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.271933    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.301096 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.301060    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8682c2-1888-4569-9b49-a12f036b968b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.368241 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.368203    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4qzf\" (UniqueName: \"kubernetes.io/projected/1a8682c2-1888-4569-9b49-a12f036b968b-kube-api-access-p4qzf\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.388419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.388383    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4qzf\" (UniqueName: \"kubernetes.io/projected/1a8682c2-1888-4569-9b49-a12f036b968b-kube-api-access-p4qzf\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-r5b9v\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:14.670161 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:14.670116    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:15.006609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:15.006575    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"]
Oct 29 22:37:15.009691 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:37:15.009658    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8682c2_1888_4569_9b49_a12f036b968b.slice/crio-254898a4eb2132b988a9e1f039c2d01533322749a074752086df40ddb7de0e76 WatchSource:0}: Error finding container 254898a4eb2132b988a9e1f039c2d01533322749a074752086df40ddb7de0e76: Status 404 returned error can't find the container with id 254898a4eb2132b988a9e1f039c2d01533322749a074752086df40ddb7de0e76
Oct 29 22:37:15.014298 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:15.014254    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:37:15.014414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:15.014374    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:37:15.014471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:15.014419    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:37:15.288269 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:15.288186    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v" event={"ID":"1a8682c2-1888-4569-9b49-a12f036b968b","Type":"ContainerStarted","Data":"5304fca4a21d1f6010aca1b3e95b822e26110ee075e68623a8423d251c37a94f"}
Oct 29 22:37:15.288269 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:15.288242    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v" event={"ID":"1a8682c2-1888-4569-9b49-a12f036b968b","Type":"ContainerStarted","Data":"254898a4eb2132b988a9e1f039c2d01533322749a074752086df40ddb7de0e76"}
Oct 29 22:37:15.337721 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:15.337649    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v" podStartSLOduration=2.337626758 podStartE2EDuration="2.337626758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:15.334551421 +0000 UTC m=+1374.245994976" watchObservedRunningTime="2026-10-29 22:37:15.337626758 +0000 UTC m=+1374.249070314"
Oct 29 22:37:15.670440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:15.670400    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:16.672971 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:16.672930    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 29 22:37:16.673393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:16.672996    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v" podUID="1a8682c2-1888-4569-9b49-a12f036b968b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 29 22:37:17.689703 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:17.689237    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 29 22:37:17.689703 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:17.689310    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v" podUID="1a8682c2-1888-4569-9b49-a12f036b968b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 29 22:37:18.690836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:18.672107    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 29 22:37:18.690836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:18.672178    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v" podUID="1a8682c2-1888-4569-9b49-a12f036b968b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 29 22:37:19.670713 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:19.670661    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 29 22:37:19.671009 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:19.670727    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v" podUID="1a8682c2-1888-4569-9b49-a12f036b968b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 29 22:37:20.032139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:20.032058    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"]
Oct 29 22:37:20.071882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:20.071847    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"]
Oct 29 22:37:20.077404 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:20.077368    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d747a8f-sptx8"]
Oct 29 22:37:20.084069 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:20.084039    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0ae17c-5xsdg"]
Oct 29 22:37:20.671592 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:20.671554    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 29 22:37:20.671945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:20.671915    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v" podUID="1a8682c2-1888-4569-9b49-a12f036b968b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 29 22:37:21.317832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:21.314611    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b44b9797-70d5-4a9e-a5bb-9b6afe189ab0" path="/var/lib/kubelet/pods/b44b9797-70d5-4a9e-a5bb-9b6afe189ab0/volumes"
Oct 29 22:37:21.317832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:21.315471    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f" path="/var/lib/kubelet/pods/dcbe0061-a4e6-4a4d-9be9-5b8a0b6c615f/volumes"
Oct 29 22:37:21.670613 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:21.670574    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 29 22:37:21.670827 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:21.670645    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v" podUID="1a8682c2-1888-4569-9b49-a12f036b968b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 29 22:37:22.674347 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:22.674311    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 29 22:37:22.677091 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:22.677049    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v" podUID="1a8682c2-1888-4569-9b49-a12f036b968b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 29 22:37:24.216673 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:24.216629    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:24.220217 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:24.220188    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:24.234170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:24.234134    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:37:28.615687 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:28.615652    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:37:32.143238 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:32.142581    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:32.263836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:32.227960    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:32.263836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:32.228849    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:37:32.263836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:32.260266    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:32.263836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:32.262899    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:32.873126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:32.859742    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:33.147577 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:33.147512    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:33.153832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:33.148198    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:37:33.227079 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:33.207257    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 22:37:33.246956 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:33.246894    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Oct 29 22:37:33.346947 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:33.346890    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:33.536881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:33.477042    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:37:33.548520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:33.508105    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:33.549471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:33.540939    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:33.804930 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:33.804834    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:37:33.805227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:33.805211    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:37:33.805398 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:33.805386    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:37:33.805577 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:33.805551    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:37:33.902835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:33.899870    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:33.902835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:33.900561    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:37:34.151893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:34.150144    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": context deadline exceeded"
Oct 29 22:37:34.236198 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:34.236142    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:34.239833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:34.239034    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:37:34.387072 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:34.387028    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:37:34.551331 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:34.551229    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:34.573859 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:34.573334    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:37:34.574557 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:34.573369    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:37:34.829509 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:34.829417    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:37:34.911031 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:34.910987    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:37:35.193037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:35.192975    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:35.268595 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:35.268270    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:35.391975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:35.391926    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:35.581532 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:35.581341    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:35.605058 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:35.604992    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:35.913952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:35.913905    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:36.194063 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:36.193947    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:36.271977 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:36.271928    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:36.399891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:36.399839    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:36.581906 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:36.579968    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:36.585852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:36.584906    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:36.618840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:36.608466    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:36.687967 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:36.687919    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:36.857013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:36.856910    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-m55mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:37:36.857268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:36.857237    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-m55mt" podUID="d166a89c-98b6-4209-b8d8-5b405090031b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:36.916837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:36.914940    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:37.203430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:37.197932    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:37.264080 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:37.264031    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:37.292410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:37.292363    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:37.408859 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:37.408437    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:37.598057 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:37.597937    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:37.621840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:37.599367    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:37:37.628308 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:37.617326    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:37:37.632840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:37.629282    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:37:37.990852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:37.987031    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:38.205954 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:38.205906    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:38.344333 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:38.344194    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:38.479834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:38.457882    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:38.604027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:38.603911    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:38.628953 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:38.628913    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:37:38.629602 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:38.629202    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:37:38.682915 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:38.682874    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9098-b14728"]
Oct 29 22:37:38.769849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:38.769021    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9098-b14728"]
Oct 29 22:37:38.769849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:38.769146    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9098-b14728"
Oct 29 22:37:38.806875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:38.796229    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xl57t\""
Oct 29 22:37:38.806875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:38.798475    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtb68\" (UniqueName: \"kubernetes.io/projected/83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e-kube-api-access-vtb68\") pod \"test-curl-9098-b14728\" (UID: \"83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e\") " pod="opendatahub/test-curl-9098-b14728"
Oct 29 22:37:39.169446 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:38.987267    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtb68\" (UniqueName: \"kubernetes.io/projected/83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e-kube-api-access-vtb68\") pod \"test-curl-9098-b14728\" (UID: \"83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e\") " pod="opendatahub/test-curl-9098-b14728"
Oct 29 22:37:39.169446 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:38.989877    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:39.169446 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:38.997847    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:37:39.169446 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:39.131700    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:39.264725 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:39.264692    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtb68\" (UniqueName: \"kubernetes.io/projected/83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e-kube-api-access-vtb68\") pod \"test-curl-9098-b14728\" (UID: \"83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e\") " pod="opendatahub/test-curl-9098-b14728"
Oct 29 22:37:39.394531 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:39.348176    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:39.462506 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:39.458885    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:39.529308 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:39.529258    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:39.618891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:39.617208    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:37:39.630008 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:39.629942    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:39.631398 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:39.631363    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:37:40.004742 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:40.003117    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:37:40.209130 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:40.207911    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:40.455837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:40.449109    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:40.461169 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:40.460105    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:40.498364 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:40.498322    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:40.662373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:40.661982    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:40.674548 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:40.673928    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:40.680269 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:40.676972    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:40.859461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:40.859185    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9098-b14728"
Oct 29 22:37:40.962050 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:37:40.929959    4559 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.694s"
Oct 29 22:37:40.985248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:40.985216    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:37:40.989962 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:40.989935    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:37:41.020091 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:41.020055    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:37:41.101057 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:41.101026    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:37:41.123853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:41.121986    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:37:41.173837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:41.149177    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:37:41.196515 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:41.191028    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:37:41.471025 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:41.470983    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9098-b14728"]
Oct 29 22:37:41.516021 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:41.515961    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9098-b14728" event={"ID":"83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e","Type":"ContainerStarted","Data":"ab5fed69fee37c4681499c193fac816f9131556dd5462973ff277acbda0210bc"}
Oct 29 22:37:42.746451 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:42.746383    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9098-b14728" event={"ID":"83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e","Type":"ContainerStarted","Data":"5aa6eba48c302a01f77c084fccffd31b468c934b605fb0b107ddb56d5f7075c2"}
Oct 29 22:37:42.774847 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:42.772159    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"]
Oct 29 22:37:42.891848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:42.879484    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9098-b14728" podStartSLOduration=4.879462252 podStartE2EDuration="4.879462252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:42.84636824 +0000 UTC m=+1401.757811793" watchObservedRunningTime="2026-10-29 22:37:42.879462252 +0000 UTC m=+1401.790905807"
Oct 29 22:37:43.167001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:43.166080    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"]
Oct 29 22:37:43.167001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:43.166299    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"
Oct 29 22:37:43.239509 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:43.238443    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-aeda8c-dockercfg-wqf6h\""
Oct 29 22:37:43.239509 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:43.238578    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-aeda8c\""
Oct 29 22:37:43.276629 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:43.276583    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5gxx\" (UniqueName: \"kubernetes.io/projected/666f8f93-509b-42a4-becd-e3b89ebe144b-kube-api-access-z5gxx\") pod \"payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st\" (UID: \"666f8f93-509b-42a4-becd-e3b89ebe144b\") " pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"
Oct 29 22:37:43.291846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:43.283181    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666f8f93-509b-42a4-becd-e3b89ebe144b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st\" (UID: \"666f8f93-509b-42a4-becd-e3b89ebe144b\") " pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"
Oct 29 22:37:43.415960 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:43.408523    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5gxx\" (UniqueName: \"kubernetes.io/projected/666f8f93-509b-42a4-becd-e3b89ebe144b-kube-api-access-z5gxx\") pod \"payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st\" (UID: \"666f8f93-509b-42a4-becd-e3b89ebe144b\") " pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"
Oct 29 22:37:43.415960 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:43.408643    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666f8f93-509b-42a4-becd-e3b89ebe144b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st\" (UID: \"666f8f93-509b-42a4-becd-e3b89ebe144b\") " pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"
Oct 29 22:37:43.415960 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:43.409583    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666f8f93-509b-42a4-becd-e3b89ebe144b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st\" (UID: \"666f8f93-509b-42a4-becd-e3b89ebe144b\") " pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"
Oct 29 22:37:43.450960 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:43.431988    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5gxx\" (UniqueName: \"kubernetes.io/projected/666f8f93-509b-42a4-becd-e3b89ebe144b-kube-api-access-z5gxx\") pod \"payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st\" (UID: \"666f8f93-509b-42a4-becd-e3b89ebe144b\") " pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"
Oct 29 22:37:43.604167 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:43.604123    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"
Oct 29 22:37:43.717109 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:37:43.717016    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:37:44.040892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:44.038283    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9098-b14728"]
Oct 29 22:37:44.065973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:44.065917    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9098-b14728"]
Oct 29 22:37:44.189292 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:44.189146    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"]
Oct 29 22:37:44.773258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:44.772927    4559 patch_prober.go:29] interesting pod/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:37:44.773258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:44.772992    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:44.798206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:44.798165    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9098-b14728" podUID="83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e" containerName="test-curl-9098-b14728" containerID="cri-o://5aa6eba48c302a01f77c084fccffd31b468c934b605fb0b107ddb56d5f7075c2" gracePeriod=2
Oct 29 22:37:44.799211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:44.799185    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st" event={"ID":"666f8f93-509b-42a4-becd-e3b89ebe144b","Type":"ContainerStarted","Data":"6d88454b805507be2fa6bcea0da8b6a38f6d734d74f92570462307e98dc19683"}
Oct 29 22:37:44.799266 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:44.799244    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st" event={"ID":"666f8f93-509b-42a4-becd-e3b89ebe144b","Type":"ContainerStarted","Data":"f59ca4b28665a3a0bb9cc50d0ee695cd13ca13d19d205352cdd7340606dd8d0a"}
Oct 29 22:37:44.799306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:44.799272    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"
Oct 29 22:37:44.871185 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:44.871060    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st" podStartSLOduration=2.871038052 podStartE2EDuration="2.871038052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:44.853994318 +0000 UTC m=+1403.765437905" watchObservedRunningTime="2026-10-29 22:37:44.871038052 +0000 UTC m=+1403.782481609"
Oct 29 22:37:44.871742 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:37:44.871121    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9098-b14728\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e" pod="opendatahub/test-curl-9098-b14728"
Oct 29 22:37:44.885289 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:37:44.885240    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9098-b14728\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e" pod="opendatahub/test-curl-9098-b14728"
Oct 29 22:37:47.170879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:47.167004    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9098-b14728"
Oct 29 22:37:47.227571 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:47.227530    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e-kube-api-access-vtb68\" (UniqueName: \"kubernetes.io/projected/83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e-kube-api-access-vtb68\") pod \"83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e\" (UID: \"83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e\") "
Oct 29 22:37:47.240453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:47.240377    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e-kube-api-access-vtb68" pod "83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e" (UID: "83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e"). InnerVolumeSpecName "kube-api-access-vtb68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:37:47.330652 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:47.330605    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtb68\" (UniqueName: \"kubernetes.io/projected/83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e-kube-api-access-vtb68\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:37:47.898909 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:47.892620    4559 generic.go:356] "Generic (PLEG): container finished" podID="83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e" containerID="5aa6eba48c302a01f77c084fccffd31b468c934b605fb0b107ddb56d5f7075c2" exitCode=137
Oct 29 22:37:47.898909 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:47.892755    4559 scope.go:122] "RemoveContainer" containerID="5aa6eba48c302a01f77c084fccffd31b468c934b605fb0b107ddb56d5f7075c2"
Oct 29 22:37:47.898909 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:47.892957    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9098-b14728"
Oct 29 22:37:48.032078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:48.020255    4559 scope.go:122] "RemoveContainer" containerID="5aa6eba48c302a01f77c084fccffd31b468c934b605fb0b107ddb56d5f7075c2"
Oct 29 22:37:48.032801 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:37:48.032689    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aa6eba48c302a01f77c084fccffd31b468c934b605fb0b107ddb56d5f7075c2\": container with ID starting with 5aa6eba48c302a01f77c084fccffd31b468c934b605fb0b107ddb56d5f7075c2 not found: ID does not exist" containerID="5aa6eba48c302a01f77c084fccffd31b468c934b605fb0b107ddb56d5f7075c2"
Oct 29 22:37:48.032801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:48.032738    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa6eba48c302a01f77c084fccffd31b468c934b605fb0b107ddb56d5f7075c2"} err="failed to get container status \"5aa6eba48c302a01f77c084fccffd31b468c934b605fb0b107ddb56d5f7075c2\": rpc error: code = NotFound desc = could not find container \"5aa6eba48c302a01f77c084fccffd31b468c934b605fb0b107ddb56d5f7075c2\": container with ID starting with 5aa6eba48c302a01f77c084fccffd31b468c934b605fb0b107ddb56d5f7075c2 not found: ID does not exist"
Oct 29 22:37:49.380486 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:49.380450    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e" path="/var/lib/kubelet/pods/83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e/volumes"
Oct 29 22:37:52.199386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:52.198969    4559 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": read tcp 10.134.0.2:50652->10.134.0.111:15021: read: connection reset by peer" start-of-body=
Oct 29 22:37:52.199386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:52.199048    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": read tcp 10.134.0.2:50652->10.134.0.111:15021: read: connection reset by peer"
Oct 29 22:37:52.871858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:52.868885    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"]
Oct 29 22:37:52.871858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:52.870227    4559 cpu_manager.go:414] "containerMap: removing container" podUID="83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e" containerName="test-curl-9098-b14728"
Oct 29 22:37:52.871858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:52.870250    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e" containerName="test-curl-9098-b14728"
Oct 29 22:37:52.871858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:52.870481    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f616e6-fbaa-4ff8-a14f-3ec7e19bed3e" containerName="test-curl-9098-b14728"
Oct 29 22:37:52.907861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:52.903234    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:52.987855 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:52.919050    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3d1777-openshift-default-dockercfg-87hwn\""
Oct 29 22:37:53.037436 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.035182    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"]
Oct 29 22:37:53.116897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.115468    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-data\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.116897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.116362    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-envoy\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.116897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.116542    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-workload-socket\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.116897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.116576    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-credential-socket\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.116897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.116608    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-podinfo\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.116897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.116678    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcx6t\" (UniqueName: \"kubernetes.io/projected/5d1bb1ea-189a-4033-b702-7fba8db86cdf-kube-api-access-kcx6t\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.119453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.118312    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-workload-certs\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.119453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.118353    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istiod-ca-cert\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.119453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.118396    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-token\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.131020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.128216    4559 generic.go:356] "Generic (PLEG): container finished" podID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerID="5fbc65ca6eaab59555eb417db20e1b9251264228558ca765d8a14f9f167efae2" exitCode=0
Oct 29 22:37:53.131020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.128336    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" event={"ID":"bf5762b8-dbb2-4967-a9c5-8a2876e913fb","Type":"ContainerDied","Data":"5fbc65ca6eaab59555eb417db20e1b9251264228558ca765d8a14f9f167efae2"}
Oct 29 22:37:53.131020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.129048    4559 scope.go:122] "RemoveContainer" containerID="5fbc65ca6eaab59555eb417db20e1b9251264228558ca765d8a14f9f167efae2"
Oct 29 22:37:53.317964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.315843    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-data\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.317964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.315979    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-envoy\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.317964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.316262    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-workload-socket\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.317964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.316308    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-credential-socket\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.317964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.316306    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-data\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.317964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.316370    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-podinfo\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.317964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.316454    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcx6t\" (UniqueName: \"kubernetes.io/projected/5d1bb1ea-189a-4033-b702-7fba8db86cdf-kube-api-access-kcx6t\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.317964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.316614    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-credential-socket\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.317964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.316646    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-workload-certs\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.317964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.316683    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istiod-ca-cert\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.317964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.316727    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-token\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.318971 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.318166    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istiod-ca-cert\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.318971 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.318731    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-workload-certs\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.319086 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.319047    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-workload-socket\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.325557 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.324914    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:37:53.325557 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.325095    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:37:53.325557 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.325192    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:37:53.338381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.338275    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-envoy\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.355736 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.355584    4559 generic.go:356] "Generic (PLEG): container finished" podID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerID="fe86c21f19f80948a127927d9d77da3248c1bd765ff7f602c4c0bd0b01f22b92" exitCode=0
Oct 29 22:37:53.355736 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.355643    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcx6t\" (UniqueName: \"kubernetes.io/projected/5d1bb1ea-189a-4033-b702-7fba8db86cdf-kube-api-access-kcx6t\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.355736 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.355678    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-podinfo\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.356371 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.356313    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-token\") pod \"e2e-api-b-3d1777-openshift-default-56f984b757-knrnv\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") " pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.364896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.359935    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" event={"ID":"2040bbf0-5596-401b-b7f6-b3fa571c4d94","Type":"ContainerDied","Data":"fe86c21f19f80948a127927d9d77da3248c1bd765ff7f602c4c0bd0b01f22b92"}
Oct 29 22:37:53.364896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.360857    4559 scope.go:122] "RemoveContainer" containerID="fe86c21f19f80948a127927d9d77da3248c1bd765ff7f602c4c0bd0b01f22b92"
Oct 29 22:37:53.366585 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.366358    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:37:53.366585 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.366462    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:37:53.366585 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.366510    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:37:53.421114 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.420716    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:53.635922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.632828    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"]
Oct 29 22:37:53.647522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.647482    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"
Oct 29 22:37:53.664667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.664041    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-39322f\""
Oct 29 22:37:53.752753 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.705882    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-39322f-dockercfg-pk65j\""
Oct 29 22:37:53.774993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.774951    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b258ed23-5da1-4feb-9b5f-a7de1f12d8b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q\" (UID: \"b258ed23-5da1-4feb-9b5f-a7de1f12d8b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"
Oct 29 22:37:53.782229 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.782184    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l54jm\" (UniqueName: \"kubernetes.io/projected/b258ed23-5da1-4feb-9b5f-a7de1f12d8b0-kube-api-access-l54jm\") pod \"payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q\" (UID: \"b258ed23-5da1-4feb-9b5f-a7de1f12d8b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"
Oct 29 22:37:53.782902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.782872    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"]
Oct 29 22:37:53.821996 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.821539    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"]
Oct 29 22:37:53.836096 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.836064    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"]
Oct 29 22:37:53.838660 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:53.838625    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"
Oct 29 22:37:54.053000 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.022329    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75chl\" (UniqueName: \"kubernetes.io/projected/6a63fc53-8f88-49ef-82ae-acde0f36fd5d-kube-api-access-75chl\") pod \"payload-processing-e2e-api-b-39322f-5c74d47cd-88276\" (UID: \"6a63fc53-8f88-49ef-82ae-acde0f36fd5d\") " pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"
Oct 29 22:37:54.053538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.053471    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l54jm\" (UniqueName: \"kubernetes.io/projected/b258ed23-5da1-4feb-9b5f-a7de1f12d8b0-kube-api-access-l54jm\") pod \"payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q\" (UID: \"b258ed23-5da1-4feb-9b5f-a7de1f12d8b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"
Oct 29 22:37:54.081239 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.081202    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a63fc53-8f88-49ef-82ae-acde0f36fd5d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-39322f-5c74d47cd-88276\" (UID: \"6a63fc53-8f88-49ef-82ae-acde0f36fd5d\") " pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"
Oct 29 22:37:54.205835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.199676    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b258ed23-5da1-4feb-9b5f-a7de1f12d8b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q\" (UID: \"b258ed23-5da1-4feb-9b5f-a7de1f12d8b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"
Oct 29 22:37:54.205835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.156533    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l54jm\" (UniqueName: \"kubernetes.io/projected/b258ed23-5da1-4feb-9b5f-a7de1f12d8b0-kube-api-access-l54jm\") pod \"payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q\" (UID: \"b258ed23-5da1-4feb-9b5f-a7de1f12d8b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"
Oct 29 22:37:54.205835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.205364    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b258ed23-5da1-4feb-9b5f-a7de1f12d8b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q\" (UID: \"b258ed23-5da1-4feb-9b5f-a7de1f12d8b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"
Oct 29 22:37:54.324463 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.324384    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"
Oct 29 22:37:54.333072 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.326972    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75chl\" (UniqueName: \"kubernetes.io/projected/6a63fc53-8f88-49ef-82ae-acde0f36fd5d-kube-api-access-75chl\") pod \"payload-processing-e2e-api-b-39322f-5c74d47cd-88276\" (UID: \"6a63fc53-8f88-49ef-82ae-acde0f36fd5d\") " pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"
Oct 29 22:37:54.333432 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.333395    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a63fc53-8f88-49ef-82ae-acde0f36fd5d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-39322f-5c74d47cd-88276\" (UID: \"6a63fc53-8f88-49ef-82ae-acde0f36fd5d\") " pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"
Oct 29 22:37:54.341709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.340733    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a63fc53-8f88-49ef-82ae-acde0f36fd5d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-39322f-5c74d47cd-88276\" (UID: \"6a63fc53-8f88-49ef-82ae-acde0f36fd5d\") " pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"
Oct 29 22:37:54.397332 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.395958    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" event={"ID":"bf5762b8-dbb2-4967-a9c5-8a2876e913fb","Type":"ContainerStarted","Data":"45bd044f8a32aeabe74230c5220b8782c00d68c898db7f9f11fd202019031dff"}
Oct 29 22:37:54.399586 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.399481    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75chl\" (UniqueName: \"kubernetes.io/projected/6a63fc53-8f88-49ef-82ae-acde0f36fd5d-kube-api-access-75chl\") pod \"payload-processing-e2e-api-b-39322f-5c74d47cd-88276\" (UID: \"6a63fc53-8f88-49ef-82ae-acde0f36fd5d\") " pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"
Oct 29 22:37:54.402829 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.400729    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" event={"ID":"2040bbf0-5596-401b-b7f6-b3fa571c4d94","Type":"ContainerStarted","Data":"16e3a9277206cd2b7f0b17cc9fbab78fcb17bee92bb2788863dea3bceb5fc985"}
Oct 29 22:37:54.429511 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.429475    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"]
Oct 29 22:37:54.430228 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:37:54.430179    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1bb1ea_189a_4033_b702_7fba8db86cdf.slice/crio-40013671cdf424ece95299c6a59da1f570ca49679562263f34940057f318b113 WatchSource:0}: Error finding container 40013671cdf424ece95299c6a59da1f570ca49679562263f34940057f318b113: Status 404 returned error can't find the container with id 40013671cdf424ece95299c6a59da1f570ca49679562263f34940057f318b113
Oct 29 22:37:54.441208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.437320    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:37:54.441208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.437538    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:37:54.441208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.437616    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:37:54.640524 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.640265    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"
Oct 29 22:37:54.772010 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:37:54.771945    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb258ed23_5da1_4feb_9b5f_a7de1f12d8b0.slice/crio-0885d6925360da9d6b85a01930f7a310e66f2830d61bdfd17d472e7c0834f5d0 WatchSource:0}: Error finding container 0885d6925360da9d6b85a01930f7a310e66f2830d61bdfd17d472e7c0834f5d0: Status 404 returned error can't find the container with id 0885d6925360da9d6b85a01930f7a310e66f2830d61bdfd17d472e7c0834f5d0
Oct 29 22:37:54.952157 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.864912    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"]
Oct 29 22:37:54.952157 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.918051    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:37:54.952157 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.951591    4559 patch_prober.go:29] interesting pod/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:37:54.952157 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:54.951643    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 29 22:37:55.226059 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:55.222503    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:37:55.237835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:55.236064    4559 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:37:55.237835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:55.236126    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 29 22:37:55.707394 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:55.707359    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"]
Oct 29 22:37:55.711647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:55.711611    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv" event={"ID":"5d1bb1ea-189a-4033-b702-7fba8db86cdf","Type":"ContainerStarted","Data":"7ffc1e613e0ea35e105c2caa1119cf09feeebafd0223e8a9b3ed0582f498cb3d"}
Oct 29 22:37:55.711911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:55.711890    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv" event={"ID":"5d1bb1ea-189a-4033-b702-7fba8db86cdf","Type":"ContainerStarted","Data":"40013671cdf424ece95299c6a59da1f570ca49679562263f34940057f318b113"}
Oct 29 22:37:55.712032 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:55.712019    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"]
Oct 29 22:37:55.712150 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:55.712135    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q" event={"ID":"b258ed23-5da1-4feb-9b5f-a7de1f12d8b0","Type":"ContainerStarted","Data":"0885d6925360da9d6b85a01930f7a310e66f2830d61bdfd17d472e7c0834f5d0"}
Oct 29 22:37:55.880852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:55.880751    4559 patch_prober.go:29] interesting pod/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:37:56.445952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:56.420174    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 29 22:37:56.445952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:56.421025    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"
Oct 29 22:37:56.504682 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:56.504641    4559 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:37:56.505013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:56.504969    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 29 22:37:56.538928 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:56.538781    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:56.564408 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:56.552646    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv" podStartSLOduration=4.552621036 podStartE2EDuration="4.552621036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:55.801406015 +0000 UTC m=+1414.712849570" watchObservedRunningTime="2026-10-29 22:37:56.552621036 +0000 UTC m=+1415.464064591"
Oct 29 22:37:56.626832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:56.609119    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:56.772715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:56.772619    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Oct 29 22:37:57.103898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:57.080249    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:57.119932 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:57.099077    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:37:57.156838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:57.149052    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"]
Oct 29 22:37:57.220760 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:57.220726    4559 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:37:57.221072 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:57.221043    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 29 22:37:57.313439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:57.309186    4559 patch_prober.go:29] interesting pod/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:37:57.313439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:57.309250    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 29 22:37:57.430488 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:57.430450    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276" event={"ID":"6a63fc53-8f88-49ef-82ae-acde0f36fd5d","Type":"ContainerStarted","Data":"c4525583becbcb9da5d2963c3c651a7959bd16b9c26c9c69ea3465344635e399"}
Oct 29 22:37:57.430738 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:57.430721    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:57.430941 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:57.430881    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q" event={"ID":"b258ed23-5da1-4feb-9b5f-a7de1f12d8b0","Type":"ContainerStarted","Data":"473e0d41728af5cae3d802ee755e9dd38fac8c0e4cbf8c38cc8bca3a1f15d6c8"}
Oct 29 22:37:57.431233 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:57.431218    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:37:57.431753 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:57.431736    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"
Oct 29 22:37:57.592839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:57.578910    4559 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:37:57.592839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:57.578977    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq" podUID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:37:57.872837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:57.871934    4559 patch_prober.go:29] interesting pod/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:37:57.872837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:57.872010    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 29 22:37:58.205878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:58.193630    4559 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:37:58.205878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:58.193728    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 29 22:37:58.502038 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:58.501957    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276" event={"ID":"6a63fc53-8f88-49ef-82ae-acde0f36fd5d","Type":"ContainerStarted","Data":"17e9c38421f411d1248ca8fb55016f52838466cb65ef8c3cbb1fe14901ba8f63"}
Oct 29 22:37:58.502200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:58.502161    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"
Oct 29 22:37:58.602892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:58.602795    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276" podStartSLOduration=5.602771307 podStartE2EDuration="5.602771307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:58.557228777 +0000 UTC m=+1417.468672334" watchObservedRunningTime="2026-10-29 22:37:58.602771307 +0000 UTC m=+1417.514214862"
Oct 29 22:37:58.676682 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:58.676620    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q" podStartSLOduration=5.676601663 podStartE2EDuration="5.676601663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:57.572197698 +0000 UTC m=+1416.483641253" watchObservedRunningTime="2026-10-29 22:37:58.676601663 +0000 UTC m=+1417.588045218"
Oct 29 22:37:58.733977 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:58.733943    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:37:58.895523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:58.892247    4559 patch_prober.go:29] interesting pod/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:37:58.895523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:58.892303    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 29 22:37:59.216374 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:59.214901    4559 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:37:59.216374 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:59.214967    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 29 22:37:59.883052 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:59.883015    4559 patch_prober.go:29] interesting pod/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:37:59.883490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:37:59.883456    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 29 22:38:00.194205 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:00.193686    4559 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:38:00.194205 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:00.193736    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 29 22:38:00.330429 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:00.330386    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"]
Oct 29 22:38:00.361837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:00.361306    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"
Oct 29 22:38:00.394118 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:00.394077    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3d1777\""
Oct 29 22:38:00.394323 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:00.394165    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3d1777-dockercfg-dmvjt\""
Oct 29 22:38:01.173087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.159653    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw" podUID="6e695e40-8026-49fb-80f3-f8c22dcf206d" containerName="payload-pre-processing" containerID="cri-o://d0e5c135f00706ba888fa57aaa81413b319a1bed968943d91039ba57190a3cbd" gracePeriod=25
Oct 29 22:38:01.272798 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.272760    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh62j\" (UniqueName: \"kubernetes.io/projected/5271c98e-3534-4fbc-8908-4ecc13444202-kube-api-access-bh62j\") pod \"payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w\" (UID: \"5271c98e-3534-4fbc-8908-4ecc13444202\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"
Oct 29 22:38:01.317839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.302624    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5271c98e-3534-4fbc-8908-4ecc13444202-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w\" (UID: \"5271c98e-3534-4fbc-8908-4ecc13444202\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"
Oct 29 22:38:01.317839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.275426    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"]
Oct 29 22:38:01.342838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.332461    4559 patch_prober.go:29] interesting pod/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:38:01.342838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.332545    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 29 22:38:01.397832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.391303    4559 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:38:01.397832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.391392    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 29 22:38:01.409873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.409754    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869" podUID="11c6a530-568d-42fa-a268-344b4fb7bd19" containerName="payload-processing" containerID="cri-o://0122ca2b3add8cafbf49e57678f10eb8cceb7bd2d436e2cce033f4975b28f423" gracePeriod=25
Oct 29 22:38:01.438897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.413328    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh62j\" (UniqueName: \"kubernetes.io/projected/5271c98e-3534-4fbc-8908-4ecc13444202-kube-api-access-bh62j\") pod \"payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w\" (UID: \"5271c98e-3534-4fbc-8908-4ecc13444202\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"
Oct 29 22:38:01.439421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.439382    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5271c98e-3534-4fbc-8908-4ecc13444202-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w\" (UID: \"5271c98e-3534-4fbc-8908-4ecc13444202\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"
Oct 29 22:38:01.507846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.444563    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5271c98e-3534-4fbc-8908-4ecc13444202-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w\" (UID: \"5271c98e-3534-4fbc-8908-4ecc13444202\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"
Oct 29 22:38:01.516611 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.516567    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh62j\" (UniqueName: \"kubernetes.io/projected/5271c98e-3534-4fbc-8908-4ecc13444202-kube-api-access-bh62j\") pod \"payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w\" (UID: \"5271c98e-3534-4fbc-8908-4ecc13444202\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"
Oct 29 22:38:01.564546 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.564491    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"]
Oct 29 22:38:01.608706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.608543    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"]
Oct 29 22:38:01.608706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.608664    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"
Oct 29 22:38:01.708944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.705197    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fff6a521-a276-49f3-a64d-5929e6cc963d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d\" (UID: \"fff6a521-a276-49f3-a64d-5929e6cc963d\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"
Oct 29 22:38:01.708944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.705465    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9kqc\" (UniqueName: \"kubernetes.io/projected/fff6a521-a276-49f3-a64d-5929e6cc963d-kube-api-access-z9kqc\") pod \"payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d\" (UID: \"fff6a521-a276-49f3-a64d-5929e6cc963d\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"
Oct 29 22:38:01.708944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.705857    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw" podUID="6e695e40-8026-49fb-80f3-f8c22dcf206d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:38:01.805526 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.805472    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"
Oct 29 22:38:01.808966 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.807276    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9kqc\" (UniqueName: \"kubernetes.io/projected/fff6a521-a276-49f3-a64d-5929e6cc963d-kube-api-access-z9kqc\") pod \"payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d\" (UID: \"fff6a521-a276-49f3-a64d-5929e6cc963d\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"
Oct 29 22:38:01.808966 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.807914    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fff6a521-a276-49f3-a64d-5929e6cc963d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d\" (UID: \"fff6a521-a276-49f3-a64d-5929e6cc963d\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"
Oct 29 22:38:01.808966 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.808902    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fff6a521-a276-49f3-a64d-5929e6cc963d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d\" (UID: \"fff6a521-a276-49f3-a64d-5929e6cc963d\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"
Oct 29 22:38:01.873908 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.873803    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9kqc\" (UniqueName: \"kubernetes.io/projected/fff6a521-a276-49f3-a64d-5929e6cc963d-kube-api-access-z9kqc\") pod \"payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d\" (UID: \"fff6a521-a276-49f3-a64d-5929e6cc963d\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"
Oct 29 22:38:01.912850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.912794    4559 patch_prober.go:29] interesting pod/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:38:01.913469 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.913435    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 29 22:38:01.928623 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:01.926989    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"
Oct 29 22:38:02.212857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:02.206523    4559 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:38:02.212857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:02.206589    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 29 22:38:02.597321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:02.596334    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"]
Oct 29 22:38:02.781001 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:38:02.780959    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff6a521_a276_49f3_a64d_5929e6cc963d.slice/crio-3790781ef2650e0baed0aed7120b0a7cc037e6954aab1794a2a3f4ecd5d2af53 WatchSource:0}: Error finding container 3790781ef2650e0baed0aed7120b0a7cc037e6954aab1794a2a3f4ecd5d2af53: Status 404 returned error can't find the container with id 3790781ef2650e0baed0aed7120b0a7cc037e6954aab1794a2a3f4ecd5d2af53
Oct 29 22:38:02.798015 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:02.797566    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"]
Oct 29 22:38:02.864057 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:02.863571    4559 patch_prober.go:29] interesting pod/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:38:02.864057 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:02.863633    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 29 22:38:03.266250 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:03.266220    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:38:03.268865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:03.267229    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:38:03.268865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:03.268269    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:38:03.524447 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:03.524416    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d" event={"ID":"fff6a521-a276-49f3-a64d-5929e6cc963d","Type":"ContainerStarted","Data":"4d905f4eefec9f8863d2f20c7c9030a4a56a45c7819c807b335caaba75663804"}
Oct 29 22:38:03.524648 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:03.524480    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d" event={"ID":"fff6a521-a276-49f3-a64d-5929e6cc963d","Type":"ContainerStarted","Data":"3790781ef2650e0baed0aed7120b0a7cc037e6954aab1794a2a3f4ecd5d2af53"}
Oct 29 22:38:03.524648 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:03.524629    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"
Oct 29 22:38:03.526516 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:03.526460    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w" event={"ID":"5271c98e-3534-4fbc-8908-4ecc13444202","Type":"ContainerStarted","Data":"3857ae3b65df82fbe436a830701d3095760006cde89559559703ff19d10a78d4"}
Oct 29 22:38:03.526516 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:03.526509    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w" event={"ID":"5271c98e-3534-4fbc-8908-4ecc13444202","Type":"ContainerStarted","Data":"072937ca45aaf7971025ab6ffa8f981c863a1039292d56a750cba9d3aa972d5b"}
Oct 29 22:38:03.526678 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:03.526660    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"
Oct 29 22:38:03.566461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:03.566357    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d" podStartSLOduration=3.566341795 podStartE2EDuration="3.566341795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:03.564279637 +0000 UTC m=+1422.475723192" watchObservedRunningTime="2026-10-29 22:38:03.566341795 +0000 UTC m=+1422.477785385"
Oct 29 22:38:03.609115 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:03.609062    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w" podStartSLOduration=3.609048182 podStartE2EDuration="3.609048182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:03.604033978 +0000 UTC m=+1422.515477534" watchObservedRunningTime="2026-10-29 22:38:03.609048182 +0000 UTC m=+1422.520491736"
Oct 29 22:38:03.895941 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:03.886492    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:38:03.907910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:03.903101    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:38:04.541740 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:04.541697    4559 generic.go:356] "Generic (PLEG): container finished" podID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerID="c4a5219786d6befd7b16d47848679d84df82024bb6f167e1c9a8ba8b1339b75b" exitCode=0
Oct 29 22:38:04.541965 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:04.541767    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq" event={"ID":"0d6a7797-f46f-45bd-a2ca-47736b909cf9","Type":"ContainerDied","Data":"c4a5219786d6befd7b16d47848679d84df82024bb6f167e1c9a8ba8b1339b75b"}
Oct 29 22:38:04.542378 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:04.542356    4559 scope.go:122] "RemoveContainer" containerID="c4a5219786d6befd7b16d47848679d84df82024bb6f167e1c9a8ba8b1339b75b"
Oct 29 22:38:04.544168 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:04.544142    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:38:04.544276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:04.544207    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:38:04.544276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:04.544239    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:38:05.554998 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:05.554958    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq" event={"ID":"0d6a7797-f46f-45bd-a2ca-47736b909cf9","Type":"ContainerStarted","Data":"fe8e13a3bb89e92f73da49ebb2f2e600ac14ba52dd600d742653a1bf6202af1e"}
Oct 29 22:38:06.109320 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:06.107993    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:38:06.109661 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:06.109628    4559 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:38:06.109720 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:06.109694    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq" podUID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 22:38:07.104687 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:07.104639    4559 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:38:07.105195 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:07.104711    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq" podUID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 22:38:07.204733 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:07.204695    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"]
Oct 29 22:38:07.205011 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:07.204979    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q" podUID="4efa9443-3039-48cf-ad64-6d36ee0e1335" containerName="istio-proxy" containerID="cri-o://e7ed50ad58560238b27737a4f2199e48f808d2aac656ba114b751327c735e8a4" gracePeriod=30
Oct 29 22:38:07.470835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:07.466951    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"]
Oct 29 22:38:07.470835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:07.467274    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn" podUID="e3bc3b1a-f61c-4835-98e5-b9efccd4470e" containerName="istio-proxy" containerID="cri-o://eb4683ce3c129c13ab9af10b5e8a3b00f6523e7c95fc7a97bb5100ad45a54a3e" gracePeriod=30
Oct 29 22:38:07.485893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:07.485865    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"]
Oct 29 22:38:07.486340 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:07.486311    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn" podUID="7ee61a67-ee9f-4311-9042-9920dea1d942" containerName="istio-proxy" containerID="cri-o://14f60b1c66e8127054548dcea75a18e55ccac104fdf5e25e7eedbdd142ff0679" gracePeriod=30
Oct 29 22:38:07.490463 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:07.490440    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"]
Oct 29 22:38:07.490679 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:07.490638    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy" containerID="cri-o://3acdf10efab044392e57975c53cce62be10ca01d7db448cb636877d281cfbaf3" gracePeriod=30
Oct 29 22:38:07.498110 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:07.498089    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"]
Oct 29 22:38:07.498414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:07.498393    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt" podUID="6167de0d-b93b-4c88-8239-2c697bbd7c7a" containerName="istio-proxy" containerID="cri-o://64705d378f0851ba2b019888b290b03fff15dda939e5290e62c89b123e0c75f0" gracePeriod=30
Oct 29 22:38:07.504454 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:07.504430    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"]
Oct 29 22:38:07.504661 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:07.504602    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf" podUID="3255ca0c-f905-4eae-bb1b-b4d06622086d" containerName="maas-api" containerID="cri-o://772784b523a66d07bbc64e105b5840ea6595bf2c6b368c73419ab8a737c7a5e2" gracePeriod=30
Oct 29 22:38:07.874898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:07.872749    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:38:07.874898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:07.872851    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:38:08.108587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:08.108543    4559 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:38:08.109228 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:08.109173    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq" podUID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 22:38:08.661472 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:08.661429    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:09.211186 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.159433    4559 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:38:09.211186 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.159533    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq" podUID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 22:38:09.211186 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.204199    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:09.230973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.230837    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:09.336224 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.336119    4559 generic.go:356] "Generic (PLEG): container finished" podID="3255ca0c-f905-4eae-bb1b-b4d06622086d" containerID="772784b523a66d07bbc64e105b5840ea6595bf2c6b368c73419ab8a737c7a5e2" exitCode=0
Oct 29 22:38:09.337948 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.337910    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:09.394127 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.394088    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:38:09.409540 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.409509    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf" event={"ID":"3255ca0c-f905-4eae-bb1b-b4d06622086d","Type":"ContainerDied","Data":"772784b523a66d07bbc64e105b5840ea6595bf2c6b368c73419ab8a737c7a5e2"}
Oct 29 22:38:09.409721 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.409556    4559 scope.go:122] "RemoveContainer" containerID="772784b523a66d07bbc64e105b5840ea6595bf2c6b368c73419ab8a737c7a5e2"
Oct 29 22:38:09.512123 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.512090    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3255ca0c-f905-4eae-bb1b-b4d06622086d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3255ca0c-f905-4eae-bb1b-b4d06622086d-maas-api-tls\") pod \"3255ca0c-f905-4eae-bb1b-b4d06622086d\" (UID: \"3255ca0c-f905-4eae-bb1b-b4d06622086d\") "
Oct 29 22:38:09.512317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.512157    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3255ca0c-f905-4eae-bb1b-b4d06622086d-kube-api-access-cp5l6\" (UniqueName: \"kubernetes.io/projected/3255ca0c-f905-4eae-bb1b-b4d06622086d-kube-api-access-cp5l6\") pod \"3255ca0c-f905-4eae-bb1b-b4d06622086d\" (UID: \"3255ca0c-f905-4eae-bb1b-b4d06622086d\") "
Oct 29 22:38:09.512317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.512197    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3255ca0c-f905-4eae-bb1b-b4d06622086d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3255ca0c-f905-4eae-bb1b-b4d06622086d-openshift-service-ca\") pod \"3255ca0c-f905-4eae-bb1b-b4d06622086d\" (UID: \"3255ca0c-f905-4eae-bb1b-b4d06622086d\") "
Oct 29 22:38:09.512573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.512549    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3255ca0c-f905-4eae-bb1b-b4d06622086d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3255ca0c-f905-4eae-bb1b-b4d06622086d-metrics-certs\") pod \"3255ca0c-f905-4eae-bb1b-b4d06622086d\" (UID: \"3255ca0c-f905-4eae-bb1b-b4d06622086d\") "
Oct 29 22:38:09.512688 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.512661    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3255ca0c-f905-4eae-bb1b-b4d06622086d-openshift-service-ca" pod "3255ca0c-f905-4eae-bb1b-b4d06622086d" (UID: "3255ca0c-f905-4eae-bb1b-b4d06622086d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:09.513306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.513281    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3255ca0c-f905-4eae-bb1b-b4d06622086d-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:09.514707 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.514684    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3255ca0c-f905-4eae-bb1b-b4d06622086d-maas-api-tls" pod "3255ca0c-f905-4eae-bb1b-b4d06622086d" (UID: "3255ca0c-f905-4eae-bb1b-b4d06622086d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:38:09.514801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.514727    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3255ca0c-f905-4eae-bb1b-b4d06622086d-kube-api-access-cp5l6" pod "3255ca0c-f905-4eae-bb1b-b4d06622086d" (UID: "3255ca0c-f905-4eae-bb1b-b4d06622086d"). InnerVolumeSpecName "kube-api-access-cp5l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:09.515069 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.515044    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3255ca0c-f905-4eae-bb1b-b4d06622086d-metrics-certs" pod "3255ca0c-f905-4eae-bb1b-b4d06622086d" (UID: "3255ca0c-f905-4eae-bb1b-b4d06622086d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:38:09.515270 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.515256    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"
Oct 29 22:38:09.741835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.657321    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3255ca0c-f905-4eae-bb1b-b4d06622086d-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:09.769911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.769877    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3255ca0c-f905-4eae-bb1b-b4d06622086d-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:09.793954 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:09.782059    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp5l6\" (UniqueName: \"kubernetes.io/projected/3255ca0c-f905-4eae-bb1b-b4d06622086d-kube-api-access-cp5l6\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:10.105131 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:10.105020    4559 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:38:10.105496 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:10.105445    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq" podUID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 22:38:10.475953 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:10.475911    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf" event={"ID":"3255ca0c-f905-4eae-bb1b-b4d06622086d","Type":"ContainerDied","Data":"7b0756a17418c6b0b8841615b8e17a64cba39f34a93d155364e56f7e52a77532"}
Oct 29 22:38:10.476696 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:10.476668    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"
Oct 29 22:38:10.554837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:10.550049    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"]
Oct 29 22:38:10.562952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:10.562902    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf"]
Oct 29 22:38:10.606995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:10.606961    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869" podUID="11c6a530-568d-42fa-a268-344b4fb7bd19" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:38:11.105486 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:11.105448    4559 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:38:11.105668 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:11.105516    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq" podUID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 22:38:11.230966 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:11.230932    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3255ca0c-f905-4eae-bb1b-b4d06622086d" path="/var/lib/kubelet/pods/3255ca0c-f905-4eae-bb1b-b4d06622086d/volumes"
Oct 29 22:38:11.448975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:11.448937    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:38:11.520152 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:11.520124    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"
Oct 29 22:38:11.664496 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:11.664451    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw" podUID="6e695e40-8026-49fb-80f3-f8c22dcf206d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:38:11.885884 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:11.885755    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1131a29d-695b9bf5f7-dkwtf" podUID="3255ca0c-f905-4eae-bb1b-b4d06622086d" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.100:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:12.170976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:12.162026    4559 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:38:12.170976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:12.162110    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq" podUID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 22:38:12.531788 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:12.531755    4559 generic.go:356] "Generic (PLEG): container finished" podID="4efa9443-3039-48cf-ad64-6d36ee0e1335" containerID="e7ed50ad58560238b27737a4f2199e48f808d2aac656ba114b751327c735e8a4" exitCode=0
Oct 29 22:38:12.532248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:12.532082    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q" event={"ID":"4efa9443-3039-48cf-ad64-6d36ee0e1335","Type":"ContainerDied","Data":"e7ed50ad58560238b27737a4f2199e48f808d2aac656ba114b751327c735e8a4"}
Oct 29 22:38:12.811919 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:12.808466    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:38:12.929761 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:12.929369    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:38:13.018873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.008581    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-workload-certs\") pod \"4efa9443-3039-48cf-ad64-6d36ee0e1335\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") "
Oct 29 22:38:13.018873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.008733    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-podinfo\") pod \"4efa9443-3039-48cf-ad64-6d36ee0e1335\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") "
Oct 29 22:38:13.018873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.008835    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-envoy\") pod \"4efa9443-3039-48cf-ad64-6d36ee0e1335\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") "
Oct 29 22:38:13.018873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.008945    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-token\" (UniqueName: \"kubernetes.io/projected/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-token\") pod \"4efa9443-3039-48cf-ad64-6d36ee0e1335\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") "
Oct 29 22:38:13.018873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.009025    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-workload-socket\") pod \"4efa9443-3039-48cf-ad64-6d36ee0e1335\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") "
Oct 29 22:38:13.018873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.009053    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-credential-socket\") pod \"4efa9443-3039-48cf-ad64-6d36ee0e1335\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") "
Oct 29 22:38:13.018873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.009229    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4efa9443-3039-48cf-ad64-6d36ee0e1335-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efa9443-3039-48cf-ad64-6d36ee0e1335-istiod-ca-cert\") pod \"4efa9443-3039-48cf-ad64-6d36ee0e1335\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") "
Oct 29 22:38:13.018873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.009258    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-data\") pod \"4efa9443-3039-48cf-ad64-6d36ee0e1335\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") "
Oct 29 22:38:13.018873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.009284    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efa9443-3039-48cf-ad64-6d36ee0e1335-kube-api-access-nmgtm\" (UniqueName: \"kubernetes.io/projected/4efa9443-3039-48cf-ad64-6d36ee0e1335-kube-api-access-nmgtm\") pod \"4efa9443-3039-48cf-ad64-6d36ee0e1335\" (UID: \"4efa9443-3039-48cf-ad64-6d36ee0e1335\") "
Oct 29 22:38:13.022228 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.020706    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-credential-socket" pod "4efa9443-3039-48cf-ad64-6d36ee0e1335" (UID: "4efa9443-3039-48cf-ad64-6d36ee0e1335"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.022228 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.021018    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-workload-socket" pod "4efa9443-3039-48cf-ad64-6d36ee0e1335" (UID: "4efa9443-3039-48cf-ad64-6d36ee0e1335"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.022228 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.021634    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.022228 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.021658    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.025860 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.025588    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-envoy" pod "4efa9443-3039-48cf-ad64-6d36ee0e1335" (UID: "4efa9443-3039-48cf-ad64-6d36ee0e1335"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.026245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.026048    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-workload-certs" pod "4efa9443-3039-48cf-ad64-6d36ee0e1335" (UID: "4efa9443-3039-48cf-ad64-6d36ee0e1335"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.026245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.026071    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efa9443-3039-48cf-ad64-6d36ee0e1335-kube-api-access-nmgtm" pod "4efa9443-3039-48cf-ad64-6d36ee0e1335" (UID: "4efa9443-3039-48cf-ad64-6d36ee0e1335"). InnerVolumeSpecName "kube-api-access-nmgtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:13.026398 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.026252    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-data" pod "4efa9443-3039-48cf-ad64-6d36ee0e1335" (UID: "4efa9443-3039-48cf-ad64-6d36ee0e1335"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.026628 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.026596    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efa9443-3039-48cf-ad64-6d36ee0e1335-istiod-ca-cert" pod "4efa9443-3039-48cf-ad64-6d36ee0e1335" (UID: "4efa9443-3039-48cf-ad64-6d36ee0e1335"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:13.034372 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.028916    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-podinfo" pod "4efa9443-3039-48cf-ad64-6d36ee0e1335" (UID: "4efa9443-3039-48cf-ad64-6d36ee0e1335"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:38:13.034372 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.031479    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-token" pod "4efa9443-3039-48cf-ad64-6d36ee0e1335" (UID: "4efa9443-3039-48cf-ad64-6d36ee0e1335"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:13.110390 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.105217    4559 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:38:13.110390 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.105281    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq" podUID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 22:38:13.125090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.122867    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istiod-ca-cert\") pod \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") "
Oct 29 22:38:13.125090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.122934    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-workload-certs\") pod \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") "
Oct 29 22:38:13.125090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.123063    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-workload-socket\") pod \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") "
Oct 29 22:38:13.125090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.123099    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-data\") pod \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") "
Oct 29 22:38:13.125090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.123217    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-podinfo\") pod \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") "
Oct 29 22:38:13.125090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.123612    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-workload-certs" pod "e3bc3b1a-f61c-4835-98e5-b9efccd4470e" (UID: "e3bc3b1a-f61c-4835-98e5-b9efccd4470e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.125090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.124165    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istiod-ca-cert" pod "e3bc3b1a-f61c-4835-98e5-b9efccd4470e" (UID: "e3bc3b1a-f61c-4835-98e5-b9efccd4470e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:13.125090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.124459    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-workload-socket" pod "e3bc3b1a-f61c-4835-98e5-b9efccd4470e" (UID: "e3bc3b1a-f61c-4835-98e5-b9efccd4470e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.125090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.124577    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-token\" (UniqueName: \"kubernetes.io/projected/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-token\") pod \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") "
Oct 29 22:38:13.125090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.124640    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-credential-socket\") pod \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") "
Oct 29 22:38:13.125090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.124669    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-kube-api-access-2q8rw\" (UniqueName: \"kubernetes.io/projected/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-kube-api-access-2q8rw\") pod \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") "
Oct 29 22:38:13.125090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.124678    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-data" pod "e3bc3b1a-f61c-4835-98e5-b9efccd4470e" (UID: "e3bc3b1a-f61c-4835-98e5-b9efccd4470e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.125090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.124940    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-envoy\") pod \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\" (UID: \"e3bc3b1a-f61c-4835-98e5-b9efccd4470e\") "
Oct 29 22:38:13.126013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.125494    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-credential-socket" pod "e3bc3b1a-f61c-4835-98e5-b9efccd4470e" (UID: "e3bc3b1a-f61c-4835-98e5-b9efccd4470e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.126580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.126418    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.126580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.126442    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.126580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.126455    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.126580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.126469    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.126580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.126482    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.126580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.126495    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efa9443-3039-48cf-ad64-6d36ee0e1335-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.126580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.126509    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.126580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.126522    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmgtm\" (UniqueName: \"kubernetes.io/projected/4efa9443-3039-48cf-ad64-6d36ee0e1335-kube-api-access-nmgtm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.126580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.126537    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efa9443-3039-48cf-ad64-6d36ee0e1335-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.126580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.126551    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.126580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.126563    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.126580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.126576    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efa9443-3039-48cf-ad64-6d36ee0e1335-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.137438 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.135927    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-envoy" pod "e3bc3b1a-f61c-4835-98e5-b9efccd4470e" (UID: "e3bc3b1a-f61c-4835-98e5-b9efccd4470e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.137438 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.136235    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-podinfo" pod "e3bc3b1a-f61c-4835-98e5-b9efccd4470e" (UID: "e3bc3b1a-f61c-4835-98e5-b9efccd4470e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:38:13.140903 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.140274    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-kube-api-access-2q8rw" pod "e3bc3b1a-f61c-4835-98e5-b9efccd4470e" (UID: "e3bc3b1a-f61c-4835-98e5-b9efccd4470e"). InnerVolumeSpecName "kube-api-access-2q8rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:13.142719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.141610    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-token" pod "e3bc3b1a-f61c-4835-98e5-b9efccd4470e" (UID: "e3bc3b1a-f61c-4835-98e5-b9efccd4470e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:13.184261 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.183995    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:38:13.186252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.186220    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:38:13.206660 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.206585    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:38:13.228440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.228401    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.228641 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.228455    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.228641 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.228479    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q8rw\" (UniqueName: \"kubernetes.io/projected/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-kube-api-access-2q8rw\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.228641 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.228506    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3b1a-f61c-4835-98e5-b9efccd4470e-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.329894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.329863    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-credential-socket\") pod \"707246aa-d09a-4f11-96f3-2778aa943e1b\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") "
Oct 29 22:38:13.330045 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.329911    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-data\") pod \"7ee61a67-ee9f-4311-9042-9920dea1d942\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") "
Oct 29 22:38:13.330045 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.329940    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-workload-certs\") pod \"7ee61a67-ee9f-4311-9042-9920dea1d942\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") "
Oct 29 22:38:13.330045 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.329965    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-envoy\") pod \"7ee61a67-ee9f-4311-9042-9920dea1d942\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") "
Oct 29 22:38:13.330045 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.330017    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istiod-ca-cert\") pod \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") "
Oct 29 22:38:13.330252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.330046    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-workload-socket\") pod \"7ee61a67-ee9f-4311-9042-9920dea1d942\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") "
Oct 29 22:38:13.330252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.330113    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-workload-socket\") pod \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") "
Oct 29 22:38:13.330252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.330148    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-podinfo\") pod \"707246aa-d09a-4f11-96f3-2778aa943e1b\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") "
Oct 29 22:38:13.330252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.330191    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-envoy\") pod \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") "
Oct 29 22:38:13.330252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.330226    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/707246aa-d09a-4f11-96f3-2778aa943e1b-kube-api-access-bt2nk\" (UniqueName: \"kubernetes.io/projected/707246aa-d09a-4f11-96f3-2778aa943e1b-kube-api-access-bt2nk\") pod \"707246aa-d09a-4f11-96f3-2778aa943e1b\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") "
Oct 29 22:38:13.330496 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.330325    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-podinfo\") pod \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") "
Oct 29 22:38:13.330496 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.330356    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/707246aa-d09a-4f11-96f3-2778aa943e1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707246aa-d09a-4f11-96f3-2778aa943e1b-istiod-ca-cert\") pod \"707246aa-d09a-4f11-96f3-2778aa943e1b\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") "
Oct 29 22:38:13.330496 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.330408    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-token\" (UniqueName: \"kubernetes.io/projected/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-token\") pod \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") "
Oct 29 22:38:13.330496 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.330474    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee61a67-ee9f-4311-9042-9920dea1d942-kube-api-access-jfvq5\" (UniqueName: \"kubernetes.io/projected/7ee61a67-ee9f-4311-9042-9920dea1d942-kube-api-access-jfvq5\") pod \"7ee61a67-ee9f-4311-9042-9920dea1d942\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") "
Oct 29 22:38:13.330693 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.330611    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-workload-certs\") pod \"707246aa-d09a-4f11-96f3-2778aa943e1b\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") "
Oct 29 22:38:13.330693 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.330644    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-workload-certs\") pod \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") "
Oct 29 22:38:13.330693 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.330673    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-data\") pod \"707246aa-d09a-4f11-96f3-2778aa943e1b\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") "
Oct 29 22:38:13.330911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.330729    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6167de0d-b93b-4c88-8239-2c697bbd7c7a-kube-api-access-tnmbn\" (UniqueName: \"kubernetes.io/projected/6167de0d-b93b-4c88-8239-2c697bbd7c7a-kube-api-access-tnmbn\") pod \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") "
Oct 29 22:38:13.330911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.330762    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-credential-socket\") pod \"7ee61a67-ee9f-4311-9042-9920dea1d942\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") "
Oct 29 22:38:13.330911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.330830    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee61a67-ee9f-4311-9042-9920dea1d942-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee61a67-ee9f-4311-9042-9920dea1d942-istiod-ca-cert\") pod \"7ee61a67-ee9f-4311-9042-9920dea1d942\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") "
Oct 29 22:38:13.330911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.330884    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-podinfo\") pod \"7ee61a67-ee9f-4311-9042-9920dea1d942\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") "
Oct 29 22:38:13.331186 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.331000    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-envoy\") pod \"707246aa-d09a-4f11-96f3-2778aa943e1b\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") "
Oct 29 22:38:13.331186 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.331039    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-data\") pod \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") "
Oct 29 22:38:13.331186 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.331066    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-token\" (UniqueName: \"kubernetes.io/projected/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-token\") pod \"707246aa-d09a-4f11-96f3-2778aa943e1b\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") "
Oct 29 22:38:13.331186 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.331110    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-token\" (UniqueName: \"kubernetes.io/projected/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-token\") pod \"7ee61a67-ee9f-4311-9042-9920dea1d942\" (UID: \"7ee61a67-ee9f-4311-9042-9920dea1d942\") "
Oct 29 22:38:13.331186 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.331147    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-workload-socket\") pod \"707246aa-d09a-4f11-96f3-2778aa943e1b\" (UID: \"707246aa-d09a-4f11-96f3-2778aa943e1b\") "
Oct 29 22:38:13.331426 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.331233    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-credential-socket\") pod \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\" (UID: \"6167de0d-b93b-4c88-8239-2c697bbd7c7a\") "
Oct 29 22:38:13.333587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.332084    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-workload-socket" pod "6167de0d-b93b-4c88-8239-2c697bbd7c7a" (UID: "6167de0d-b93b-4c88-8239-2c697bbd7c7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.333587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.332280    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-workload-socket" pod "7ee61a67-ee9f-4311-9042-9920dea1d942" (UID: "7ee61a67-ee9f-4311-9042-9920dea1d942"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.333587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.332287    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.333587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.332481    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-data" pod "7ee61a67-ee9f-4311-9042-9920dea1d942" (UID: "7ee61a67-ee9f-4311-9042-9920dea1d942"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.333587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.332701    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-credential-socket" pod "707246aa-d09a-4f11-96f3-2778aa943e1b" (UID: "707246aa-d09a-4f11-96f3-2778aa943e1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.333587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.332944    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-credential-socket" pod "7ee61a67-ee9f-4311-9042-9920dea1d942" (UID: "7ee61a67-ee9f-4311-9042-9920dea1d942"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.333587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.333291    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-workload-certs" pod "7ee61a67-ee9f-4311-9042-9920dea1d942" (UID: "7ee61a67-ee9f-4311-9042-9920dea1d942"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.333587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.333295    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707246aa-d09a-4f11-96f3-2778aa943e1b-istiod-ca-cert" pod "707246aa-d09a-4f11-96f3-2778aa943e1b" (UID: "707246aa-d09a-4f11-96f3-2778aa943e1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:13.335227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.335199    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istiod-ca-cert" pod "6167de0d-b93b-4c88-8239-2c697bbd7c7a" (UID: "6167de0d-b93b-4c88-8239-2c697bbd7c7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:13.335363 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.335325    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-workload-socket" pod "707246aa-d09a-4f11-96f3-2778aa943e1b" (UID: "707246aa-d09a-4f11-96f3-2778aa943e1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.335855 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.335830    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-credential-socket" pod "6167de0d-b93b-4c88-8239-2c697bbd7c7a" (UID: "6167de0d-b93b-4c88-8239-2c697bbd7c7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.336300 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.336279    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee61a67-ee9f-4311-9042-9920dea1d942-istiod-ca-cert" pod "7ee61a67-ee9f-4311-9042-9920dea1d942" (UID: "7ee61a67-ee9f-4311-9042-9920dea1d942"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:13.336687 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.336668    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-workload-certs" pod "6167de0d-b93b-4c88-8239-2c697bbd7c7a" (UID: "6167de0d-b93b-4c88-8239-2c697bbd7c7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.338206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.338178    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-data" pod "707246aa-d09a-4f11-96f3-2778aa943e1b" (UID: "707246aa-d09a-4f11-96f3-2778aa943e1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.338206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.338194    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-workload-certs" pod "707246aa-d09a-4f11-96f3-2778aa943e1b" (UID: "707246aa-d09a-4f11-96f3-2778aa943e1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.340836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.340691    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-envoy" pod "707246aa-d09a-4f11-96f3-2778aa943e1b" (UID: "707246aa-d09a-4f11-96f3-2778aa943e1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.343607 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.343559    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-podinfo" pod "6167de0d-b93b-4c88-8239-2c697bbd7c7a" (UID: "6167de0d-b93b-4c88-8239-2c697bbd7c7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:38:13.353921 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.353852    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-envoy" pod "7ee61a67-ee9f-4311-9042-9920dea1d942" (UID: "7ee61a67-ee9f-4311-9042-9920dea1d942"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.370786 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.370525    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-token" pod "6167de0d-b93b-4c88-8239-2c697bbd7c7a" (UID: "6167de0d-b93b-4c88-8239-2c697bbd7c7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:13.370786 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.370635    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-token" pod "707246aa-d09a-4f11-96f3-2778aa943e1b" (UID: "707246aa-d09a-4f11-96f3-2778aa943e1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:13.370786 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.370703    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-token" pod "7ee61a67-ee9f-4311-9042-9920dea1d942" (UID: "7ee61a67-ee9f-4311-9042-9920dea1d942"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:13.370786 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.370763    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee61a67-ee9f-4311-9042-9920dea1d942-kube-api-access-jfvq5" pod "7ee61a67-ee9f-4311-9042-9920dea1d942" (UID: "7ee61a67-ee9f-4311-9042-9920dea1d942"). InnerVolumeSpecName "kube-api-access-jfvq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:13.371330 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.371306    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-podinfo" pod "7ee61a67-ee9f-4311-9042-9920dea1d942" (UID: "7ee61a67-ee9f-4311-9042-9920dea1d942"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:38:13.373005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.372965    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-envoy" pod "6167de0d-b93b-4c88-8239-2c697bbd7c7a" (UID: "6167de0d-b93b-4c88-8239-2c697bbd7c7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.378192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.378160    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707246aa-d09a-4f11-96f3-2778aa943e1b-kube-api-access-bt2nk" pod "707246aa-d09a-4f11-96f3-2778aa943e1b" (UID: "707246aa-d09a-4f11-96f3-2778aa943e1b"). InnerVolumeSpecName "kube-api-access-bt2nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:13.378380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.378341    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-data" pod "6167de0d-b93b-4c88-8239-2c697bbd7c7a" (UID: "6167de0d-b93b-4c88-8239-2c697bbd7c7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:13.380255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.380217    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6167de0d-b93b-4c88-8239-2c697bbd7c7a-kube-api-access-tnmbn" pod "6167de0d-b93b-4c88-8239-2c697bbd7c7a" (UID: "6167de0d-b93b-4c88-8239-2c697bbd7c7a"). InnerVolumeSpecName "kube-api-access-tnmbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:13.384751 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.384714    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-podinfo" pod "707246aa-d09a-4f11-96f3-2778aa943e1b" (UID: "707246aa-d09a-4f11-96f3-2778aa943e1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:38:13.454753 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454717    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.454753 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454755    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454769    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454784    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnmbn\" (UniqueName: \"kubernetes.io/projected/6167de0d-b93b-4c88-8239-2c697bbd7c7a-kube-api-access-tnmbn\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454798    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454828    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee61a67-ee9f-4311-9042-9920dea1d942-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454843    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454854    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454866    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454878    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454891    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454903    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454915    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454927    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707246aa-d09a-4f11-96f3-2778aa943e1b-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454942    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454955    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454966    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454979    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.454991    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee61a67-ee9f-4311-9042-9920dea1d942-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.455005    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707246aa-d09a-4f11-96f3-2778aa943e1b-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.455019    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.455032    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt2nk\" (UniqueName: \"kubernetes.io/projected/707246aa-d09a-4f11-96f3-2778aa943e1b-kube-api-access-bt2nk\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.455045    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.455058    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707246aa-d09a-4f11-96f3-2778aa943e1b-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.455070    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6167de0d-b93b-4c88-8239-2c697bbd7c7a-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.455729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.455082    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfvq5\" (UniqueName: \"kubernetes.io/projected/7ee61a67-ee9f-4311-9042-9920dea1d942-kube-api-access-jfvq5\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.542162 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.542122    4559 generic.go:356] "Generic (PLEG): container finished" podID="e3bc3b1a-f61c-4835-98e5-b9efccd4470e" containerID="eb4683ce3c129c13ab9af10b5e8a3b00f6523e7c95fc7a97bb5100ad45a54a3e" exitCode=0
Oct 29 22:38:13.542628 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.542198    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"
Oct 29 22:38:13.542628 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.542210    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn" event={"ID":"e3bc3b1a-f61c-4835-98e5-b9efccd4470e","Type":"ContainerDied","Data":"eb4683ce3c129c13ab9af10b5e8a3b00f6523e7c95fc7a97bb5100ad45a54a3e"}
Oct 29 22:38:13.542628 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.542281    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn" event={"ID":"e3bc3b1a-f61c-4835-98e5-b9efccd4470e","Type":"ContainerDied","Data":"67848abb75153432414e4774ae7884d15e5b89ab1da00a82ca893f4a7f7c2aae"}
Oct 29 22:38:13.542628 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.542311    4559 scope.go:122] "RemoveContainer" containerID="eb4683ce3c129c13ab9af10b5e8a3b00f6523e7c95fc7a97bb5100ad45a54a3e"
Oct 29 22:38:13.544315 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.544287    4559 generic.go:356] "Generic (PLEG): container finished" podID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerID="3acdf10efab044392e57975c53cce62be10ca01d7db448cb636877d281cfbaf3" exitCode=0
Oct 29 22:38:13.544433 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.544415    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" event={"ID":"707246aa-d09a-4f11-96f3-2778aa943e1b","Type":"ContainerDied","Data":"3acdf10efab044392e57975c53cce62be10ca01d7db448cb636877d281cfbaf3"}
Oct 29 22:38:13.544488 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.544463    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7" event={"ID":"707246aa-d09a-4f11-96f3-2778aa943e1b","Type":"ContainerDied","Data":"956db9e631d786f605d2277400e8e14365c817c75b898a7d6083aa39db689d49"}
Oct 29 22:38:13.544566 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.544551    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"
Oct 29 22:38:13.552386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.552350    4559 generic.go:356] "Generic (PLEG): container finished" podID="6167de0d-b93b-4c88-8239-2c697bbd7c7a" containerID="64705d378f0851ba2b019888b290b03fff15dda939e5290e62c89b123e0c75f0" exitCode=0
Oct 29 22:38:13.552528 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.552487    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt" event={"ID":"6167de0d-b93b-4c88-8239-2c697bbd7c7a","Type":"ContainerDied","Data":"64705d378f0851ba2b019888b290b03fff15dda939e5290e62c89b123e0c75f0"}
Oct 29 22:38:13.552603 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.552537    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt" event={"ID":"6167de0d-b93b-4c88-8239-2c697bbd7c7a","Type":"ContainerDied","Data":"c85c9759c283b9917c05ba26c729bc6740eaae9fa3339a7f841dd869ea9b0b63"}
Oct 29 22:38:13.552656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.552646    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"
Oct 29 22:38:13.585987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.584068    4559 scope.go:122] "RemoveContainer" containerID="eb4683ce3c129c13ab9af10b5e8a3b00f6523e7c95fc7a97bb5100ad45a54a3e"
Oct 29 22:38:13.594088 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:38:13.593093    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4683ce3c129c13ab9af10b5e8a3b00f6523e7c95fc7a97bb5100ad45a54a3e\": container with ID starting with eb4683ce3c129c13ab9af10b5e8a3b00f6523e7c95fc7a97bb5100ad45a54a3e not found: ID does not exist" containerID="eb4683ce3c129c13ab9af10b5e8a3b00f6523e7c95fc7a97bb5100ad45a54a3e"
Oct 29 22:38:13.594088 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.593243    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4683ce3c129c13ab9af10b5e8a3b00f6523e7c95fc7a97bb5100ad45a54a3e"} err="failed to get container status \"eb4683ce3c129c13ab9af10b5e8a3b00f6523e7c95fc7a97bb5100ad45a54a3e\": rpc error: code = NotFound desc = could not find container \"eb4683ce3c129c13ab9af10b5e8a3b00f6523e7c95fc7a97bb5100ad45a54a3e\": container with ID starting with eb4683ce3c129c13ab9af10b5e8a3b00f6523e7c95fc7a97bb5100ad45a54a3e not found: ID does not exist"
Oct 29 22:38:13.594088 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.593274    4559 scope.go:122] "RemoveContainer" containerID="3acdf10efab044392e57975c53cce62be10ca01d7db448cb636877d281cfbaf3"
Oct 29 22:38:13.595137 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.594735    4559 generic.go:356] "Generic (PLEG): container finished" podID="7ee61a67-ee9f-4311-9042-9920dea1d942" containerID="14f60b1c66e8127054548dcea75a18e55ccac104fdf5e25e7eedbdd142ff0679" exitCode=0
Oct 29 22:38:13.595137 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.594922    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn" event={"ID":"7ee61a67-ee9f-4311-9042-9920dea1d942","Type":"ContainerDied","Data":"14f60b1c66e8127054548dcea75a18e55ccac104fdf5e25e7eedbdd142ff0679"}
Oct 29 22:38:13.595137 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.594972    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn" event={"ID":"7ee61a67-ee9f-4311-9042-9920dea1d942","Type":"ContainerDied","Data":"8b6b7e612c834cd15e03d3c81d9fa319f438bbe9687d30f64625aa154beba04d"}
Oct 29 22:38:13.595137 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.595065    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"
Oct 29 22:38:13.607420 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.602989    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"
Oct 29 22:38:13.607420 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.602794    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q" event={"ID":"4efa9443-3039-48cf-ad64-6d36ee0e1335","Type":"ContainerDied","Data":"2bd091ec9211b534addad83c8a906c0bb7a9a5b32141682003bfa0b830df3253"}
Oct 29 22:38:13.613442 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.613361    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"]
Oct 29 22:38:13.622057 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.622030    4559 scope.go:122] "RemoveContainer" containerID="3acdf10efab044392e57975c53cce62be10ca01d7db448cb636877d281cfbaf3"
Oct 29 22:38:13.622438 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:38:13.622408    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3acdf10efab044392e57975c53cce62be10ca01d7db448cb636877d281cfbaf3\": container with ID starting with 3acdf10efab044392e57975c53cce62be10ca01d7db448cb636877d281cfbaf3 not found: ID does not exist" containerID="3acdf10efab044392e57975c53cce62be10ca01d7db448cb636877d281cfbaf3"
Oct 29 22:38:13.622547 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.622449    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3acdf10efab044392e57975c53cce62be10ca01d7db448cb636877d281cfbaf3"} err="failed to get container status \"3acdf10efab044392e57975c53cce62be10ca01d7db448cb636877d281cfbaf3\": rpc error: code = NotFound desc = could not find container \"3acdf10efab044392e57975c53cce62be10ca01d7db448cb636877d281cfbaf3\": container with ID starting with 3acdf10efab044392e57975c53cce62be10ca01d7db448cb636877d281cfbaf3 not found: ID does not exist"
Oct 29 22:38:13.622547 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.622476    4559 scope.go:122] "RemoveContainer" containerID="64705d378f0851ba2b019888b290b03fff15dda939e5290e62c89b123e0c75f0"
Oct 29 22:38:13.624340 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.624313    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-vjfcn"]
Oct 29 22:38:13.633513 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.633492    4559 scope.go:122] "RemoveContainer" containerID="64705d378f0851ba2b019888b290b03fff15dda939e5290e62c89b123e0c75f0"
Oct 29 22:38:13.634042 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:38:13.633890    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64705d378f0851ba2b019888b290b03fff15dda939e5290e62c89b123e0c75f0\": container with ID starting with 64705d378f0851ba2b019888b290b03fff15dda939e5290e62c89b123e0c75f0 not found: ID does not exist" containerID="64705d378f0851ba2b019888b290b03fff15dda939e5290e62c89b123e0c75f0"
Oct 29 22:38:13.634042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.633926    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64705d378f0851ba2b019888b290b03fff15dda939e5290e62c89b123e0c75f0"} err="failed to get container status \"64705d378f0851ba2b019888b290b03fff15dda939e5290e62c89b123e0c75f0\": rpc error: code = NotFound desc = could not find container \"64705d378f0851ba2b019888b290b03fff15dda939e5290e62c89b123e0c75f0\": container with ID starting with 64705d378f0851ba2b019888b290b03fff15dda939e5290e62c89b123e0c75f0 not found: ID does not exist"
Oct 29 22:38:13.634042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.633950    4559 scope.go:122] "RemoveContainer" containerID="14f60b1c66e8127054548dcea75a18e55ccac104fdf5e25e7eedbdd142ff0679"
Oct 29 22:38:13.648790 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.648765    4559 scope.go:122] "RemoveContainer" containerID="14f60b1c66e8127054548dcea75a18e55ccac104fdf5e25e7eedbdd142ff0679"
Oct 29 22:38:13.649162 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:38:13.649131    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f60b1c66e8127054548dcea75a18e55ccac104fdf5e25e7eedbdd142ff0679\": container with ID starting with 14f60b1c66e8127054548dcea75a18e55ccac104fdf5e25e7eedbdd142ff0679 not found: ID does not exist" containerID="14f60b1c66e8127054548dcea75a18e55ccac104fdf5e25e7eedbdd142ff0679"
Oct 29 22:38:13.649315 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.649172    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f60b1c66e8127054548dcea75a18e55ccac104fdf5e25e7eedbdd142ff0679"} err="failed to get container status \"14f60b1c66e8127054548dcea75a18e55ccac104fdf5e25e7eedbdd142ff0679\": rpc error: code = NotFound desc = could not find container \"14f60b1c66e8127054548dcea75a18e55ccac104fdf5e25e7eedbdd142ff0679\": container with ID starting with 14f60b1c66e8127054548dcea75a18e55ccac104fdf5e25e7eedbdd142ff0679 not found: ID does not exist"
Oct 29 22:38:13.649315 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.649196    4559 scope.go:122] "RemoveContainer" containerID="e7ed50ad58560238b27737a4f2199e48f808d2aac656ba114b751327c735e8a4"
Oct 29 22:38:13.704948 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.692883    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"]
Oct 29 22:38:13.749721 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.744423    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-dd4x7"]
Oct 29 22:38:13.749721 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.748994    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"]
Oct 29 22:38:13.832923 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.829175    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8gswt"]
Oct 29 22:38:13.872589 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.872487    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"]
Oct 29 22:38:13.888672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.888527    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-cb9wn"]
Oct 29 22:38:13.892324 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.892267    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"]
Oct 29 22:38:13.893735 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:38:13.893689    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:38:13.897112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:13.897085    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-q5z2q"]
Oct 29 22:38:14.109701 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:14.109670    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:38:14.110930 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:14.110913    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:38:14.545726 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:14.545695    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"
Oct 29 22:38:14.546243 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:14.546151    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"
Oct 29 22:38:15.246965 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:15.246926    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4efa9443-3039-48cf-ad64-6d36ee0e1335" path="/var/lib/kubelet/pods/4efa9443-3039-48cf-ad64-6d36ee0e1335/volumes"
Oct 29 22:38:15.247640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:15.247621    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6167de0d-b93b-4c88-8239-2c697bbd7c7a" path="/var/lib/kubelet/pods/6167de0d-b93b-4c88-8239-2c697bbd7c7a/volumes"
Oct 29 22:38:15.248144 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:15.248126    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" path="/var/lib/kubelet/pods/707246aa-d09a-4f11-96f3-2778aa943e1b/volumes"
Oct 29 22:38:15.248993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:15.248976    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee61a67-ee9f-4311-9042-9920dea1d942" path="/var/lib/kubelet/pods/7ee61a67-ee9f-4311-9042-9920dea1d942/volumes"
Oct 29 22:38:15.249424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:15.249409    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3bc3b1a-f61c-4835-98e5-b9efccd4470e" path="/var/lib/kubelet/pods/e3bc3b1a-f61c-4835-98e5-b9efccd4470e/volumes"
Oct 29 22:38:15.466795 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:15.466762    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"]
Oct 29 22:38:15.783836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:15.775698    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq" podUID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerName="istio-proxy" containerID="cri-o://fe8e13a3bb89e92f73da49ebb2f2e600ac14ba52dd600d742653a1bf6202af1e" gracePeriod=30
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.943921    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"]
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945229    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e3bc3b1a-f61c-4835-98e5-b9efccd4470e" containerName="istio-proxy"
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945251    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3bc3b1a-f61c-4835-98e5-b9efccd4470e" containerName="istio-proxy"
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945298    4559 cpu_manager.go:414] "containerMap: removing container" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy"
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945305    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy"
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945335    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7ee61a67-ee9f-4311-9042-9920dea1d942" containerName="istio-proxy"
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945343    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee61a67-ee9f-4311-9042-9920dea1d942" containerName="istio-proxy"
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945365    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4efa9443-3039-48cf-ad64-6d36ee0e1335" containerName="istio-proxy"
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945373    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efa9443-3039-48cf-ad64-6d36ee0e1335" containerName="istio-proxy"
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945400    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6167de0d-b93b-4c88-8239-2c697bbd7c7a" containerName="istio-proxy"
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945407    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6167de0d-b93b-4c88-8239-2c697bbd7c7a" containerName="istio-proxy"
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945419    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3255ca0c-f905-4eae-bb1b-b4d06622086d" containerName="maas-api"
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945425    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3255ca0c-f905-4eae-bb1b-b4d06622086d" containerName="maas-api"
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945572    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3255ca0c-f905-4eae-bb1b-b4d06622086d" containerName="maas-api"
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945583    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3bc3b1a-f61c-4835-98e5-b9efccd4470e" containerName="istio-proxy"
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945597    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="707246aa-d09a-4f11-96f3-2778aa943e1b" containerName="istio-proxy"
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945608    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee61a67-ee9f-4311-9042-9920dea1d942" containerName="istio-proxy"
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945619    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efa9443-3039-48cf-ad64-6d36ee0e1335" containerName="istio-proxy"
Oct 29 22:38:16.961836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:16.945631    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6167de0d-b93b-4c88-8239-2c697bbd7c7a" containerName="istio-proxy"
Oct 29 22:38:17.140065 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.115096    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"]
Oct 29 22:38:17.140065 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.115253    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.228406 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.228306    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4d3168-openshift-default-dockercfg-88hm9\""
Oct 29 22:38:17.438196 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.438153    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-podinfo\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.440043 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.440011    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-credential-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.440324 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.440301    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ljkb\" (UniqueName: \"kubernetes.io/projected/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-kube-api-access-8ljkb\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.440592 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.440563    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-token\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.440922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.440905    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-workload-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.441233 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.441215    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-envoy\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.442892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.442800    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istiod-ca-cert\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.443201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.443182    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-workload-certs\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.443410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.443392    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-data\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.546686 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.546571    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-credential-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.547560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.547517    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljkb\" (UniqueName: \"kubernetes.io/projected/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-kube-api-access-8ljkb\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.547963 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.547944    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-token\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.558985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.558948    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-workload-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.559287 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.559267    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-envoy\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.559478 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.559459    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istiod-ca-cert\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.559683 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.559665    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-workload-certs\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.559839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.559803    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-data\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.560059 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.560045    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-podinfo\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.561761 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.561736    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-data\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.563096 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.562158    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-workload-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.563228 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.562701    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-workload-certs\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.563442 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.563422    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istiod-ca-cert\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.563668 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.563611    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-credential-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.574724 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.574458    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-podinfo\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.574915 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.574890    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljkb\" (UniqueName: \"kubernetes.io/projected/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-kube-api-access-8ljkb\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.576220 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.576174    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-envoy\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.581973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.581216    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-token\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:17.680255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:17.680221    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:18.034952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:18.034917    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"]
Oct 29 22:38:18.039880 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:38:18.036660    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158bafa6_a6ed_49d4_8ac2_2722c7be5efb.slice/crio-139e6b8a09e34018a7150367c795c7df714153ed4e4df76820b20e5522ea8390 WatchSource:0}: Error finding container 139e6b8a09e34018a7150367c795c7df714153ed4e4df76820b20e5522ea8390: Status 404 returned error can't find the container with id 139e6b8a09e34018a7150367c795c7df714153ed4e4df76820b20e5522ea8390
Oct 29 22:38:18.040578 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:18.040539    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:38:18.040645 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:18.040629    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:38:18.040700 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:18.040672    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:38:18.303929 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:18.288495    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:38:18.813719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:18.813667    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm" event={"ID":"158bafa6-a6ed-49d4-8ac2-2722c7be5efb","Type":"ContainerStarted","Data":"f230b8fda8326c414997183af033d54402c09d6b13c91bb439562b079482c444"}
Oct 29 22:38:18.813919 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:18.813734    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm" event={"ID":"158bafa6-a6ed-49d4-8ac2-2722c7be5efb","Type":"ContainerStarted","Data":"139e6b8a09e34018a7150367c795c7df714153ed4e4df76820b20e5522ea8390"}
Oct 29 22:38:18.873878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:18.873802    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm" podStartSLOduration=2.8737840390000002 podStartE2EDuration="2.873784039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:18.870950987 +0000 UTC m=+1437.782394557" watchObservedRunningTime="2026-10-29 22:38:18.873784039 +0000 UTC m=+1437.785227592"
Oct 29 22:38:19.681208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:19.681168    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:19.687733 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:19.687701    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:19.917840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:19.917171    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:19.920849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:19.919075    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:38:20.606927 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:20.606887    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869" podUID="11c6a530-568d-42fa-a268-344b4fb7bd19" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:38:20.932430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:20.932388    4559 generic.go:356] "Generic (PLEG): container finished" podID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerID="fe8e13a3bb89e92f73da49ebb2f2e600ac14ba52dd600d742653a1bf6202af1e" exitCode=0
Oct 29 22:38:20.933680 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:20.933640    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq" event={"ID":"0d6a7797-f46f-45bd-a2ca-47736b909cf9","Type":"ContainerDied","Data":"fe8e13a3bb89e92f73da49ebb2f2e600ac14ba52dd600d742653a1bf6202af1e"}
Oct 29 22:38:20.933828 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:20.933706    4559 scope.go:122] "RemoveContainer" containerID="c4a5219786d6befd7b16d47848679d84df82024bb6f167e1c9a8ba8b1339b75b"
Oct 29 22:38:21.101902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.098359    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:38:21.319332 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.319260    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-credential-socket\") pod \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") "
Oct 29 22:38:21.319478 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.319339    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-data\") pod \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") "
Oct 29 22:38:21.319478 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.319425    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-envoy\") pod \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") "
Oct 29 22:38:21.319577 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.319536    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-podinfo\") pod \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") "
Oct 29 22:38:21.319627 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.319548    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-credential-socket" pod "0d6a7797-f46f-45bd-a2ca-47736b909cf9" (UID: "0d6a7797-f46f-45bd-a2ca-47736b909cf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:21.319672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.319620    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-data" pod "0d6a7797-f46f-45bd-a2ca-47736b909cf9" (UID: "0d6a7797-f46f-45bd-a2ca-47736b909cf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:21.319725 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.319667    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-token\" (UniqueName: \"kubernetes.io/projected/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-token\") pod \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") "
Oct 29 22:38:21.319725 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.319700    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-workload-socket\") pod \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") "
Oct 29 22:38:21.319881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.319856    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6a7797-f46f-45bd-a2ca-47736b909cf9-kube-api-access-j42hv\" (UniqueName: \"kubernetes.io/projected/0d6a7797-f46f-45bd-a2ca-47736b909cf9-kube-api-access-j42hv\") pod \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") "
Oct 29 22:38:21.320088 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.319930    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istiod-ca-cert\") pod \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") "
Oct 29 22:38:21.320088 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.319987    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-workload-certs\") pod \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\" (UID: \"0d6a7797-f46f-45bd-a2ca-47736b909cf9\") "
Oct 29 22:38:21.320373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.320293    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-workload-certs" pod "0d6a7797-f46f-45bd-a2ca-47736b909cf9" (UID: "0d6a7797-f46f-45bd-a2ca-47736b909cf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:21.320479 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.320450    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istiod-ca-cert" pod "0d6a7797-f46f-45bd-a2ca-47736b909cf9" (UID: "0d6a7797-f46f-45bd-a2ca-47736b909cf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:21.321076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.321049    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-workload-socket" pod "0d6a7797-f46f-45bd-a2ca-47736b909cf9" (UID: "0d6a7797-f46f-45bd-a2ca-47736b909cf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:21.321184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.321167    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:21.321251 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.321190    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:21.321251 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.321203    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:21.321251 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.321218    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:21.321875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.321848    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-envoy" pod "0d6a7797-f46f-45bd-a2ca-47736b909cf9" (UID: "0d6a7797-f46f-45bd-a2ca-47736b909cf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:21.322009 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.321992    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-podinfo" pod "0d6a7797-f46f-45bd-a2ca-47736b909cf9" (UID: "0d6a7797-f46f-45bd-a2ca-47736b909cf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:38:21.322292 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.322275    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6a7797-f46f-45bd-a2ca-47736b909cf9-kube-api-access-j42hv" pod "0d6a7797-f46f-45bd-a2ca-47736b909cf9" (UID: "0d6a7797-f46f-45bd-a2ca-47736b909cf9"). InnerVolumeSpecName "kube-api-access-j42hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:21.322381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.322363    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-token" pod "0d6a7797-f46f-45bd-a2ca-47736b909cf9" (UID: "0d6a7797-f46f-45bd-a2ca-47736b909cf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:21.423883 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.423856    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:21.423883 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.423881    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:21.424052 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.423891    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j42hv\" (UniqueName: \"kubernetes.io/projected/0d6a7797-f46f-45bd-a2ca-47736b909cf9-kube-api-access-j42hv\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:21.424052 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.423900    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:21.424052 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.423914    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6a7797-f46f-45bd-a2ca-47736b909cf9-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:21.606646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.606615    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"
Oct 29 22:38:21.718442 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.718416    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"
Oct 29 22:38:21.729945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.729919    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e695e40-8026-49fb-80f3-f8c22dcf206d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e695e40-8026-49fb-80f3-f8c22dcf206d-plugins-config-volume\") pod \"6e695e40-8026-49fb-80f3-f8c22dcf206d\" (UID: \"6e695e40-8026-49fb-80f3-f8c22dcf206d\") "
Oct 29 22:38:21.730108 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.730041    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e695e40-8026-49fb-80f3-f8c22dcf206d-kube-api-access-4k6x4\" (UniqueName: \"kubernetes.io/projected/6e695e40-8026-49fb-80f3-f8c22dcf206d-kube-api-access-4k6x4\") pod \"6e695e40-8026-49fb-80f3-f8c22dcf206d\" (UID: \"6e695e40-8026-49fb-80f3-f8c22dcf206d\") "
Oct 29 22:38:21.730351 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.730325    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e695e40-8026-49fb-80f3-f8c22dcf206d-plugins-config-volume" pod "6e695e40-8026-49fb-80f3-f8c22dcf206d" (UID: "6e695e40-8026-49fb-80f3-f8c22dcf206d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:21.730939 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.730913    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e695e40-8026-49fb-80f3-f8c22dcf206d-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:21.732380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.732356    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e695e40-8026-49fb-80f3-f8c22dcf206d-kube-api-access-4k6x4" pod "6e695e40-8026-49fb-80f3-f8c22dcf206d" (UID: "6e695e40-8026-49fb-80f3-f8c22dcf206d"). InnerVolumeSpecName "kube-api-access-4k6x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:21.832709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.832666    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c6a530-568d-42fa-a268-344b4fb7bd19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c6a530-568d-42fa-a268-344b4fb7bd19-plugins-config-volume\") pod \"11c6a530-568d-42fa-a268-344b4fb7bd19\" (UID: \"11c6a530-568d-42fa-a268-344b4fb7bd19\") "
Oct 29 22:38:21.832923 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.832791    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c6a530-568d-42fa-a268-344b4fb7bd19-kube-api-access-vg6pp\" (UniqueName: \"kubernetes.io/projected/11c6a530-568d-42fa-a268-344b4fb7bd19-kube-api-access-vg6pp\") pod \"11c6a530-568d-42fa-a268-344b4fb7bd19\" (UID: \"11c6a530-568d-42fa-a268-344b4fb7bd19\") "
Oct 29 22:38:21.833201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.833129    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c6a530-568d-42fa-a268-344b4fb7bd19-plugins-config-volume" pod "11c6a530-568d-42fa-a268-344b4fb7bd19" (UID: "11c6a530-568d-42fa-a268-344b4fb7bd19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:21.835213 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.833779    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k6x4\" (UniqueName: \"kubernetes.io/projected/6e695e40-8026-49fb-80f3-f8c22dcf206d-kube-api-access-4k6x4\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:21.835213 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.833857    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c6a530-568d-42fa-a268-344b4fb7bd19-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:21.836683 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.836398    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c6a530-568d-42fa-a268-344b4fb7bd19-kube-api-access-vg6pp" pod "11c6a530-568d-42fa-a268-344b4fb7bd19" (UID: "11c6a530-568d-42fa-a268-344b4fb7bd19"). InnerVolumeSpecName "kube-api-access-vg6pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:21.887178 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.887146    4559 scope.go:122] "RemoveContainer" containerID="c57bc7d4bec46ad2b3c78557e276ec076647c1cf7e9ac89cd64c053c21afae08"
Oct 29 22:38:21.906081 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.906055    4559 scope.go:122] "RemoveContainer" containerID="020710e28adc289961941ac338a8cac45e4168136eea0d1aae2421f3e826392d"
Oct 29 22:38:21.918946 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.918787    4559 scope.go:122] "RemoveContainer" containerID="5e8a305d44751437be93a4d590627ed2c4f131c0f488356af653d7724607065c"
Oct 29 22:38:21.929484 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.929460    4559 scope.go:122] "RemoveContainer" containerID="9a7a64e09170525f0eb3e1b109dc5f04a955f6f7c82d9c6c9d4117e51a3fa95e"
Oct 29 22:38:21.936064 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.936034    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg6pp\" (UniqueName: \"kubernetes.io/projected/11c6a530-568d-42fa-a268-344b4fb7bd19-kube-api-access-vg6pp\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:38:21.945819 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.945768    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq" event={"ID":"0d6a7797-f46f-45bd-a2ca-47736b909cf9","Type":"ContainerDied","Data":"76e43c18770c3528c806096308fdb4251239b630ec7e574da819935454ec4896"}
Oct 29 22:38:21.946130 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.945804    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"
Oct 29 22:38:21.946130 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.945911    4559 scope.go:122] "RemoveContainer" containerID="fe8e13a3bb89e92f73da49ebb2f2e600ac14ba52dd600d742653a1bf6202af1e"
Oct 29 22:38:21.950116 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.950087    4559 generic.go:356] "Generic (PLEG): container finished" podID="6e695e40-8026-49fb-80f3-f8c22dcf206d" containerID="d0e5c135f00706ba888fa57aaa81413b319a1bed968943d91039ba57190a3cbd" exitCode=0
Oct 29 22:38:21.950213 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.950165    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"
Oct 29 22:38:21.950213 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.950175    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw" event={"ID":"6e695e40-8026-49fb-80f3-f8c22dcf206d","Type":"ContainerDied","Data":"d0e5c135f00706ba888fa57aaa81413b319a1bed968943d91039ba57190a3cbd"}
Oct 29 22:38:21.950289 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.950223    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw" event={"ID":"6e695e40-8026-49fb-80f3-f8c22dcf206d","Type":"ContainerDied","Data":"8b0a33856c189278a29348a8d03041ee4605d051bb04d4cf924ea33553e6dd69"}
Oct 29 22:38:21.951637 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.951613    4559 generic.go:356] "Generic (PLEG): container finished" podID="11c6a530-568d-42fa-a268-344b4fb7bd19" containerID="0122ca2b3add8cafbf49e57678f10eb8cceb7bd2d436e2cce033f4975b28f423" exitCode=0
Oct 29 22:38:21.951755 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.951682    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"
Oct 29 22:38:21.951755 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.951701    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869" event={"ID":"11c6a530-568d-42fa-a268-344b4fb7bd19","Type":"ContainerDied","Data":"0122ca2b3add8cafbf49e57678f10eb8cceb7bd2d436e2cce033f4975b28f423"}
Oct 29 22:38:21.951755 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.951744    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869" event={"ID":"11c6a530-568d-42fa-a268-344b4fb7bd19","Type":"ContainerDied","Data":"a49760b90759cc000e9ee7f9d3fe1a14046b9e22c1f7ad7aa2778b0792e992b6"}
Oct 29 22:38:21.969315 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.969240    4559 scope.go:122] "RemoveContainer" containerID="d0e5c135f00706ba888fa57aaa81413b319a1bed968943d91039ba57190a3cbd"
Oct 29 22:38:21.985562 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.985465    4559 scope.go:122] "RemoveContainer" containerID="d0e5c135f00706ba888fa57aaa81413b319a1bed968943d91039ba57190a3cbd"
Oct 29 22:38:21.985929 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:38:21.985898    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e5c135f00706ba888fa57aaa81413b319a1bed968943d91039ba57190a3cbd\": container with ID starting with d0e5c135f00706ba888fa57aaa81413b319a1bed968943d91039ba57190a3cbd not found: ID does not exist" containerID="d0e5c135f00706ba888fa57aaa81413b319a1bed968943d91039ba57190a3cbd"
Oct 29 22:38:21.986043 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.985939    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e5c135f00706ba888fa57aaa81413b319a1bed968943d91039ba57190a3cbd"} err="failed to get container status \"d0e5c135f00706ba888fa57aaa81413b319a1bed968943d91039ba57190a3cbd\": rpc error: code = NotFound desc = could not find container \"d0e5c135f00706ba888fa57aaa81413b319a1bed968943d91039ba57190a3cbd\": container with ID starting with d0e5c135f00706ba888fa57aaa81413b319a1bed968943d91039ba57190a3cbd not found: ID does not exist"
Oct 29 22:38:21.986043 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:21.985960    4559 scope.go:122] "RemoveContainer" containerID="0122ca2b3add8cafbf49e57678f10eb8cceb7bd2d436e2cce033f4975b28f423"
Oct 29 22:38:22.000873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:22.000836    4559 scope.go:122] "RemoveContainer" containerID="0122ca2b3add8cafbf49e57678f10eb8cceb7bd2d436e2cce033f4975b28f423"
Oct 29 22:38:22.001254 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:38:22.001218    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0122ca2b3add8cafbf49e57678f10eb8cceb7bd2d436e2cce033f4975b28f423\": container with ID starting with 0122ca2b3add8cafbf49e57678f10eb8cceb7bd2d436e2cce033f4975b28f423 not found: ID does not exist" containerID="0122ca2b3add8cafbf49e57678f10eb8cceb7bd2d436e2cce033f4975b28f423"
Oct 29 22:38:22.001467 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:22.001252    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0122ca2b3add8cafbf49e57678f10eb8cceb7bd2d436e2cce033f4975b28f423"} err="failed to get container status \"0122ca2b3add8cafbf49e57678f10eb8cceb7bd2d436e2cce033f4975b28f423\": rpc error: code = NotFound desc = could not find container \"0122ca2b3add8cafbf49e57678f10eb8cceb7bd2d436e2cce033f4975b28f423\": container with ID starting with 0122ca2b3add8cafbf49e57678f10eb8cceb7bd2d436e2cce033f4975b28f423 not found: ID does not exist"
Oct 29 22:38:22.020538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:22.020118    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"]
Oct 29 22:38:22.024567 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:22.024320    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-xlqfw"]
Oct 29 22:38:22.061468 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:22.061436    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"]
Oct 29 22:38:22.067793 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:22.067762    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-c9869"]
Oct 29 22:38:22.114865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:22.114760    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"]
Oct 29 22:38:22.121424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:22.121393    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-fmfnq"]
Oct 29 22:38:23.059310 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.059268    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"]
Oct 29 22:38:23.060366 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.060339    4559 cpu_manager.go:414] "containerMap: removing container" podUID="11c6a530-568d-42fa-a268-344b4fb7bd19" containerName="payload-processing"
Oct 29 22:38:23.060366 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.060364    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c6a530-568d-42fa-a268-344b4fb7bd19" containerName="payload-processing"
Oct 29 22:38:23.060542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.060382    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6e695e40-8026-49fb-80f3-f8c22dcf206d" containerName="payload-pre-processing"
Oct 29 22:38:23.060542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.060390    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e695e40-8026-49fb-80f3-f8c22dcf206d" containerName="payload-pre-processing"
Oct 29 22:38:23.060542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.060405    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerName="istio-proxy"
Oct 29 22:38:23.060542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.060411    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerName="istio-proxy"
Oct 29 22:38:23.060542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.060429    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerName="istio-proxy"
Oct 29 22:38:23.060542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.060437    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerName="istio-proxy"
Oct 29 22:38:23.060852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.060693    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c6a530-568d-42fa-a268-344b4fb7bd19" containerName="payload-processing"
Oct 29 22:38:23.060852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.060710    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e695e40-8026-49fb-80f3-f8c22dcf206d" containerName="payload-pre-processing"
Oct 29 22:38:23.060852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.060720    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerName="istio-proxy"
Oct 29 22:38:23.060852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.060730    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" containerName="istio-proxy"
Oct 29 22:38:23.066694 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.066671    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:38:23.095330 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.095297    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"]
Oct 29 22:38:23.110854 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.101878    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-39322f-serving-cert\""
Oct 29 22:38:23.161587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.159009    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt29s\" (UniqueName: \"kubernetes.io/projected/ccce4424-021a-44ce-aa08-c1d46bf935a5-kube-api-access-jt29s\") pod \"maas-api-e2e-api-b-39322f-595467c98-dvbw6\" (UID: \"ccce4424-021a-44ce-aa08-c1d46bf935a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:38:23.161587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.159220    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccce4424-021a-44ce-aa08-c1d46bf935a5-metrics-certs\") pod \"maas-api-e2e-api-b-39322f-595467c98-dvbw6\" (UID: \"ccce4424-021a-44ce-aa08-c1d46bf935a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:38:23.161587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.159309    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccce4424-021a-44ce-aa08-c1d46bf935a5-maas-api-tls\") pod \"maas-api-e2e-api-b-39322f-595467c98-dvbw6\" (UID: \"ccce4424-021a-44ce-aa08-c1d46bf935a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:38:23.161587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.159357    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccce4424-021a-44ce-aa08-c1d46bf935a5-openshift-service-ca\") pod \"maas-api-e2e-api-b-39322f-595467c98-dvbw6\" (UID: \"ccce4424-021a-44ce-aa08-c1d46bf935a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:38:23.234045 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.234011    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6a7797-f46f-45bd-a2ca-47736b909cf9" path="/var/lib/kubelet/pods/0d6a7797-f46f-45bd-a2ca-47736b909cf9/volumes"
Oct 29 22:38:23.234631 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.234613    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c6a530-568d-42fa-a268-344b4fb7bd19" path="/var/lib/kubelet/pods/11c6a530-568d-42fa-a268-344b4fb7bd19/volumes"
Oct 29 22:38:23.235165 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.235148    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e695e40-8026-49fb-80f3-f8c22dcf206d" path="/var/lib/kubelet/pods/6e695e40-8026-49fb-80f3-f8c22dcf206d/volumes"
Oct 29 22:38:23.260477 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.260443    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccce4424-021a-44ce-aa08-c1d46bf935a5-maas-api-tls\") pod \"maas-api-e2e-api-b-39322f-595467c98-dvbw6\" (UID: \"ccce4424-021a-44ce-aa08-c1d46bf935a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:38:23.260700 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.260493    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccce4424-021a-44ce-aa08-c1d46bf935a5-openshift-service-ca\") pod \"maas-api-e2e-api-b-39322f-595467c98-dvbw6\" (UID: \"ccce4424-021a-44ce-aa08-c1d46bf935a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:38:23.260700 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.260633    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt29s\" (UniqueName: \"kubernetes.io/projected/ccce4424-021a-44ce-aa08-c1d46bf935a5-kube-api-access-jt29s\") pod \"maas-api-e2e-api-b-39322f-595467c98-dvbw6\" (UID: \"ccce4424-021a-44ce-aa08-c1d46bf935a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:38:23.260700 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:38:23.260635    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-serving-cert: secret "maas-api-e2e-api-b-39322f-serving-cert" not found
Oct 29 22:38:23.260922 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:38:23.260712    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccce4424-021a-44ce-aa08-c1d46bf935a5-maas-api-tls podName:ccce4424-021a-44ce-aa08-c1d46bf935a5 nodeName:}" failed. No retries permitted until 2026-10-29 22:38:23.760691616 +0000 UTC m=+1442.672135151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ccce4424-021a-44ce-aa08-c1d46bf935a5-maas-api-tls") pod "maas-api-e2e-api-b-39322f-595467c98-dvbw6" (UID: "ccce4424-021a-44ce-aa08-c1d46bf935a5") : secret "maas-api-e2e-api-b-39322f-serving-cert" not found
Oct 29 22:38:23.260983 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.260918    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccce4424-021a-44ce-aa08-c1d46bf935a5-metrics-certs\") pod \"maas-api-e2e-api-b-39322f-595467c98-dvbw6\" (UID: \"ccce4424-021a-44ce-aa08-c1d46bf935a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:38:23.261365 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.261343    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccce4424-021a-44ce-aa08-c1d46bf935a5-openshift-service-ca\") pod \"maas-api-e2e-api-b-39322f-595467c98-dvbw6\" (UID: \"ccce4424-021a-44ce-aa08-c1d46bf935a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:38:23.263887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.263858    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccce4424-021a-44ce-aa08-c1d46bf935a5-metrics-certs\") pod \"maas-api-e2e-api-b-39322f-595467c98-dvbw6\" (UID: \"ccce4424-021a-44ce-aa08-c1d46bf935a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:38:23.287874 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.287843    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt29s\" (UniqueName: \"kubernetes.io/projected/ccce4424-021a-44ce-aa08-c1d46bf935a5-kube-api-access-jt29s\") pod \"maas-api-e2e-api-b-39322f-595467c98-dvbw6\" (UID: \"ccce4424-021a-44ce-aa08-c1d46bf935a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:38:23.408212 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.408176    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"]
Oct 29 22:38:23.414419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.414393    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"
Oct 29 22:38:23.435358 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.435312    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4d3168\""
Oct 29 22:38:23.435564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.435498    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4d3168-dockercfg-gg2wh\""
Oct 29 22:38:23.439863 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.439645    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"]
Oct 29 22:38:23.580301 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.580010    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdd9l\" (UniqueName: \"kubernetes.io/projected/b1912dca-8e71-4a6c-ac38-934bd5f4a8df-kube-api-access-pdd9l\") pod \"payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k\" (UID: \"b1912dca-8e71-4a6c-ac38-934bd5f4a8df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"
Oct 29 22:38:23.582043 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.580744    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1912dca-8e71-4a6c-ac38-934bd5f4a8df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k\" (UID: \"b1912dca-8e71-4a6c-ac38-934bd5f4a8df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"
Oct 29 22:38:23.601997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.601955    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"]
Oct 29 22:38:23.676886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:23.672176    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"
Oct 29 22:38:24.060886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.058553    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"]
Oct 29 22:38:24.079096 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.079059    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdd9l\" (UniqueName: \"kubernetes.io/projected/b1912dca-8e71-4a6c-ac38-934bd5f4a8df-kube-api-access-pdd9l\") pod \"payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k\" (UID: \"b1912dca-8e71-4a6c-ac38-934bd5f4a8df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"
Oct 29 22:38:24.083048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.083014    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbmk7\" (UniqueName: \"kubernetes.io/projected/bcd744cd-3522-40f3-9ef9-857fb1ab36e9-kube-api-access-sbmk7\") pod \"payload-processing-e2e-inf-a-4d3168-d94579868-m6l26\" (UID: \"bcd744cd-3522-40f3-9ef9-857fb1ab36e9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"
Oct 29 22:38:24.083406 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.083383    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1912dca-8e71-4a6c-ac38-934bd5f4a8df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k\" (UID: \"b1912dca-8e71-4a6c-ac38-934bd5f4a8df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"
Oct 29 22:38:24.083872 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.083600    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccce4424-021a-44ce-aa08-c1d46bf935a5-maas-api-tls\") pod \"maas-api-e2e-api-b-39322f-595467c98-dvbw6\" (UID: \"ccce4424-021a-44ce-aa08-c1d46bf935a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:38:24.084159 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.084139    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcd744cd-3522-40f3-9ef9-857fb1ab36e9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4d3168-d94579868-m6l26\" (UID: \"bcd744cd-3522-40f3-9ef9-857fb1ab36e9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"
Oct 29 22:38:24.120110 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.118579    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1912dca-8e71-4a6c-ac38-934bd5f4a8df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k\" (UID: \"b1912dca-8e71-4a6c-ac38-934bd5f4a8df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"
Oct 29 22:38:24.123629 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.123594    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccce4424-021a-44ce-aa08-c1d46bf935a5-maas-api-tls\") pod \"maas-api-e2e-api-b-39322f-595467c98-dvbw6\" (UID: \"ccce4424-021a-44ce-aa08-c1d46bf935a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:38:24.146899 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.146632    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdd9l\" (UniqueName: \"kubernetes.io/projected/b1912dca-8e71-4a6c-ac38-934bd5f4a8df-kube-api-access-pdd9l\") pod \"payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k\" (UID: \"b1912dca-8e71-4a6c-ac38-934bd5f4a8df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"
Oct 29 22:38:24.218949 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.218898    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbmk7\" (UniqueName: \"kubernetes.io/projected/bcd744cd-3522-40f3-9ef9-857fb1ab36e9-kube-api-access-sbmk7\") pod \"payload-processing-e2e-inf-a-4d3168-d94579868-m6l26\" (UID: \"bcd744cd-3522-40f3-9ef9-857fb1ab36e9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"
Oct 29 22:38:24.219182 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.219164    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcd744cd-3522-40f3-9ef9-857fb1ab36e9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4d3168-d94579868-m6l26\" (UID: \"bcd744cd-3522-40f3-9ef9-857fb1ab36e9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"
Oct 29 22:38:24.219915 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.219888    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcd744cd-3522-40f3-9ef9-857fb1ab36e9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4d3168-d94579868-m6l26\" (UID: \"bcd744cd-3522-40f3-9ef9-857fb1ab36e9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"
Oct 29 22:38:24.238862 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.238786    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbmk7\" (UniqueName: \"kubernetes.io/projected/bcd744cd-3522-40f3-9ef9-857fb1ab36e9-kube-api-access-sbmk7\") pod \"payload-processing-e2e-inf-a-4d3168-d94579868-m6l26\" (UID: \"bcd744cd-3522-40f3-9ef9-857fb1ab36e9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"
Oct 29 22:38:24.284018 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.283978    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:38:24.330067 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.329964    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"
Oct 29 22:38:24.506844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.493708    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"
Oct 29 22:38:24.707131 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.706901    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"]
Oct 29 22:38:24.754839 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:38:24.754598    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccce4424_021a_44ce_aa08_c1d46bf935a5.slice/crio-bc6918afb7a86ff498a1550f2aa3a7f0d40c51b884a21cec04d361f410ae8e37 WatchSource:0}: Error finding container bc6918afb7a86ff498a1550f2aa3a7f0d40c51b884a21cec04d361f410ae8e37: Status 404 returned error can't find the container with id bc6918afb7a86ff498a1550f2aa3a7f0d40c51b884a21cec04d361f410ae8e37
Oct 29 22:38:24.776906 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:24.758539    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:38:25.196508 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:25.196401    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"]
Oct 29 22:38:25.227162 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:25.216157    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"]
Oct 29 22:38:25.250486 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:25.250451    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26" event={"ID":"bcd744cd-3522-40f3-9ef9-857fb1ab36e9","Type":"ContainerStarted","Data":"271348d47686fa1937361e8f463c488a43f8c2c68a05150e6e1057a9a138f470"}
Oct 29 22:38:25.250674 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:25.250651    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k" event={"ID":"b1912dca-8e71-4a6c-ac38-934bd5f4a8df","Type":"ContainerStarted","Data":"cd721eb8d8e31e01fa939c65d6eca77b8e9f5f5b682c8741d5c7081383730a79"}
Oct 29 22:38:25.252126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:25.252097    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6" event={"ID":"ccce4424-021a-44ce-aa08-c1d46bf935a5","Type":"ContainerStarted","Data":"bc6918afb7a86ff498a1550f2aa3a7f0d40c51b884a21cec04d361f410ae8e37"}
Oct 29 22:38:26.265471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:26.265431    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26" event={"ID":"bcd744cd-3522-40f3-9ef9-857fb1ab36e9","Type":"ContainerStarted","Data":"dc102aff2c38a8f7790d22e14c4cbca22851ab2d711650fbb3c72972bcb75b91"}
Oct 29 22:38:26.265998 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:26.265631    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"
Oct 29 22:38:26.267397 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:26.267367    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k" event={"ID":"b1912dca-8e71-4a6c-ac38-934bd5f4a8df","Type":"ContainerStarted","Data":"d5d86c226fb1102a3cefe2bdcb8751ce4d6c2a4cbdb7ffc7165ed7372e0c260b"}
Oct 29 22:38:26.267555 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:26.267476    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"
Oct 29 22:38:26.269580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:26.269548    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6" event={"ID":"ccce4424-021a-44ce-aa08-c1d46bf935a5","Type":"ContainerStarted","Data":"9089218d78dc5dcdbdd09e1d2710a866c0386cbc679133899e2cf5e8c80144a4"}
Oct 29 22:38:26.269701 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:26.269604    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:38:26.309149 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:26.309099    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26" podStartSLOduration=3.309083484 podStartE2EDuration="3.309083484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:26.30479327 +0000 UTC m=+1445.216236826" watchObservedRunningTime="2026-10-29 22:38:26.309083484 +0000 UTC m=+1445.220527042"
Oct 29 22:38:26.346823 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:26.346743    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k" podStartSLOduration=3.346725504 podStartE2EDuration="3.346725504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:26.343636434 +0000 UTC m=+1445.255079989" watchObservedRunningTime="2026-10-29 22:38:26.346725504 +0000 UTC m=+1445.258169059"
Oct 29 22:38:26.387966 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:26.387914    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6" podStartSLOduration=2.882749398 podStartE2EDuration="3.38790142s" totalImagesPullingTime="505.152022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:26.38463388 +0000 UTC m=+1445.296077435" watchObservedRunningTime="2026-10-29 22:38:26.38790142 +0000 UTC m=+1445.299344974"
Oct 29 22:38:29.653501 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:29.653464    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"]
Oct 29 22:38:29.670737 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:29.660576    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:29.696923 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:29.696886    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"]
Oct 29 22:38:29.701997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:29.701965    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c1750b-openshift-default-dockercfg-c7hg9\""
Oct 29 22:38:29.804931 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:29.797160    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istiod-ca-cert\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:29.804931 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:29.797246    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-workload-socket\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:29.932893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:29.924395    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-envoy\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:29.932893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:29.924584    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-data\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:29.946835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:29.941158    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctfkh\" (UniqueName: \"kubernetes.io/projected/71bc892b-1fac-4c55-a4c4-493bf662a9ae-kube-api-access-ctfkh\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:29.946835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:29.941317    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-credential-socket\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:29.946835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:29.941560    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-workload-certs\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:29.946835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:29.941700    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-podinfo\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:29.946835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:29.941847    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-token\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.048836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.045193    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-workload-certs\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.048836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.045307    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-podinfo\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.048836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.045382    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-token\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.048836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.045542    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istiod-ca-cert\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.048836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.045633    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-workload-socket\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.048836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.045708    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-envoy\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.048836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.045837    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-data\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.048836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.046027    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctfkh\" (UniqueName: \"kubernetes.io/projected/71bc892b-1fac-4c55-a4c4-493bf662a9ae-kube-api-access-ctfkh\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.048836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.046119    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-credential-socket\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.048836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.047284    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-credential-socket\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.049457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.049143    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-workload-certs\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.061892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.051185    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istiod-ca-cert\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.061892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.051627    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-data\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.061892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.055401    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-podinfo\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.061892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.055623    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-workload-socket\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.128457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.128417    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-token\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.154895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.151986    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"]
Oct 29 22:38:30.163837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.159420    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:38:30.163837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.162473    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-envoy\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.220074 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.209368    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctfkh\" (UniqueName: \"kubernetes.io/projected/71bc892b-1fac-4c55-a4c4-493bf662a9ae-kube-api-access-ctfkh\") pod \"e2e-inf-a-c1750b-openshift-default-85d9759655-956xt\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") " pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.258921 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.258634    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3d1777-serving-cert\""
Oct 29 22:38:30.333947 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.305801    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"]
Oct 29 22:38:30.333947 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.323377    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:30.354328 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.354200    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-maas-api-tls\") pod \"maas-api-e2e-api-b-3d1777-774f978df4-74jq6\" (UID: \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:38:30.354328 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.354264    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4q7t\" (UniqueName: \"kubernetes.io/projected/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-kube-api-access-t4q7t\") pod \"maas-api-e2e-api-b-3d1777-774f978df4-74jq6\" (UID: \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:38:30.355672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.354699    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-openshift-service-ca\") pod \"maas-api-e2e-api-b-3d1777-774f978df4-74jq6\" (UID: \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:38:30.355672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.355077    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-metrics-certs\") pod \"maas-api-e2e-api-b-3d1777-774f978df4-74jq6\" (UID: \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:38:30.462661 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.462588    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-metrics-certs\") pod \"maas-api-e2e-api-b-3d1777-774f978df4-74jq6\" (UID: \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:38:30.463066 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.463042    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-maas-api-tls\") pod \"maas-api-e2e-api-b-3d1777-774f978df4-74jq6\" (UID: \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:38:30.463304 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.463244    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4q7t\" (UniqueName: \"kubernetes.io/projected/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-kube-api-access-t4q7t\") pod \"maas-api-e2e-api-b-3d1777-774f978df4-74jq6\" (UID: \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:38:30.463581 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.463562    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-openshift-service-ca\") pod \"maas-api-e2e-api-b-3d1777-774f978df4-74jq6\" (UID: \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:38:30.464630 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.464569    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-openshift-service-ca\") pod \"maas-api-e2e-api-b-3d1777-774f978df4-74jq6\" (UID: \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:38:30.475560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.475425    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-maas-api-tls\") pod \"maas-api-e2e-api-b-3d1777-774f978df4-74jq6\" (UID: \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:38:30.481216 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.481137    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-metrics-certs\") pod \"maas-api-e2e-api-b-3d1777-774f978df4-74jq6\" (UID: \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:38:30.546192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.546156    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4q7t\" (UniqueName: \"kubernetes.io/projected/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-kube-api-access-t4q7t\") pod \"maas-api-e2e-api-b-3d1777-774f978df4-74jq6\" (UID: \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:38:30.644640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.644594    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:38:30.798605 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.798568    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"]
Oct 29 22:38:30.801718 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:38:30.801688    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bc892b_1fac_4c55_a4c4_493bf662a9ae.slice/crio-dc3fc7959fb04792ae4bdb35327075ddc84659da79f5af376c71af5cc1843899 WatchSource:0}: Error finding container dc3fc7959fb04792ae4bdb35327075ddc84659da79f5af376c71af5cc1843899: Status 404 returned error can't find the container with id dc3fc7959fb04792ae4bdb35327075ddc84659da79f5af376c71af5cc1843899
Oct 29 22:38:30.805801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.805759    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:38:30.805900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.805872    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:38:30.805935 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:30.805919    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:38:31.059093 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:31.059059    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"]
Oct 29 22:38:31.403893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:31.401213    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6" event={"ID":"2e5ef4a7-965a-4695-834d-37c4e8cf0beb","Type":"ContainerStarted","Data":"5a846dd590ad4ce98a912721aac51104ffe0a926fab95658e91b6cd842e15332"}
Oct 29 22:38:31.405847 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:31.405798    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt" event={"ID":"71bc892b-1fac-4c55-a4c4-493bf662a9ae","Type":"ContainerStarted","Data":"92c0f28d138be4dfb0bfb677c31b3358b72fc95c64844dfb794afc6fa804130a"}
Oct 29 22:38:31.406107 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:31.406073    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt" event={"ID":"71bc892b-1fac-4c55-a4c4-493bf662a9ae","Type":"ContainerStarted","Data":"dc3fc7959fb04792ae4bdb35327075ddc84659da79f5af376c71af5cc1843899"}
Oct 29 22:38:31.578057 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:31.560299    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt" podStartSLOduration=2.560277376 podStartE2EDuration="2.560277376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:31.468317289 +0000 UTC m=+1450.379760846" watchObservedRunningTime="2026-10-29 22:38:31.560277376 +0000 UTC m=+1450.471720933"
Oct 29 22:38:32.300464 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:32.299021    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:38:32.328616 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:32.328585    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:32.361318 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:32.359734    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:32.594869 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:32.594740    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6" event={"ID":"2e5ef4a7-965a-4695-834d-37c4e8cf0beb","Type":"ContainerStarted","Data":"fdb0d29e8f9b44d0e5dba075599f17f6761da55457d8bd20435786cc01f1b4c2"}
Oct 29 22:38:32.595130 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:32.595110    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:32.595277 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:32.595265    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:38:32.602161 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:32.602136    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:38:32.651217 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:32.651165    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6" podStartSLOduration=3.14165442 podStartE2EDuration="3.651148712s" totalImagesPullingTime="509.494292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:32.647548701 +0000 UTC m=+1451.558992261" watchObservedRunningTime="2026-10-29 22:38:32.651148712 +0000 UTC m=+1451.562592265"
Oct 29 22:38:37.285303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:37.285272    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"
Oct 29 22:38:37.286637 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:37.286615    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"
Oct 29 22:38:39.735725 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:39.735695    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:38:43.696542 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:38:43.696504    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:38:52.735654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:52.735618    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"]
Oct 29 22:38:52.736523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:52.736308    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82" podUID="6a3bfdf3-5fca-44b9-88d7-bf525139ee93" containerName="istio-proxy" containerID="cri-o://1c153840f8535885d6955996c97c2fc506a2450fff700e6c8ed7ecf1be7fc0e8" gracePeriod=30
Oct 29 22:38:52.749647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:52.749598    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"]
Oct 29 22:38:52.749940 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:52.749910    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t" podUID="62ebc209-264e-49db-87fd-ca9c789386b4" containerName="istio-proxy" containerID="cri-o://c4fe2ada8cd77fbef664dac65577b1068589f03db9641ae83e3ab5adddfde0c1" gracePeriod=30
Oct 29 22:38:52.754299 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:52.754270    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"]
Oct 29 22:38:52.754466 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:52.754445    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv" podUID="4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" containerName="istio-proxy" containerID="cri-o://b908886771c48e6baeeb8200680778da8bec333b878c5ffd52ffac00bfb9532a" gracePeriod=30
Oct 29 22:38:58.282344 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:58.281910    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:58.385873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:58.374876    4559 generic.go:356] "Generic (PLEG): container finished" podID="6a3bfdf3-5fca-44b9-88d7-bf525139ee93" containerID="1c153840f8535885d6955996c97c2fc506a2450fff700e6c8ed7ecf1be7fc0e8" exitCode=0
Oct 29 22:38:58.385873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:58.374976    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82" event={"ID":"6a3bfdf3-5fca-44b9-88d7-bf525139ee93","Type":"ContainerDied","Data":"1c153840f8535885d6955996c97c2fc506a2450fff700e6c8ed7ecf1be7fc0e8"}
Oct 29 22:38:58.390076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:58.390039    4559 generic.go:356] "Generic (PLEG): container finished" podID="4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" containerID="b908886771c48e6baeeb8200680778da8bec333b878c5ffd52ffac00bfb9532a" exitCode=0
Oct 29 22:38:58.390406 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:58.390379    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv" event={"ID":"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518","Type":"ContainerDied","Data":"b908886771c48e6baeeb8200680778da8bec333b878c5ffd52ffac00bfb9532a"}
Oct 29 22:38:58.401842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:58.401785    4559 generic.go:356] "Generic (PLEG): container finished" podID="62ebc209-264e-49db-87fd-ca9c789386b4" containerID="c4fe2ada8cd77fbef664dac65577b1068589f03db9641ae83e3ab5adddfde0c1" exitCode=0
Oct 29 22:38:58.402278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:58.402252    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t" event={"ID":"62ebc209-264e-49db-87fd-ca9c789386b4","Type":"ContainerDied","Data":"c4fe2ada8cd77fbef664dac65577b1068589f03db9641ae83e3ab5adddfde0c1"}
Oct 29 22:38:58.710860 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:58.682967    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:58.710860 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:58.683130    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:59.055839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:58.980544    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:38:59.128853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:59.117945    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:59.249056 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:59.230645    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"]
Oct 29 22:38:59.249842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:59.249380    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:59.289939 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:59.289892    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:59.445539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:59.436953    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:38:59.704895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:59.697107    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:59.847510 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:59.705506    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:59.878619 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:59.878574    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:38:59.878948 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:38:59.878898    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv" podUID="4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:39:00.110129 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:00.094292    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:00.373358 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:00.373240    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:00.477087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:00.477032    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:00.569837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:00.553105    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:39:00.569837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:00.559634    4559 patch_prober.go:29] interesting pod/metrics-server-58fdf66b86-ht9bn container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.14:10250/livez\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:39:00.569837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:00.559704    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-58fdf66b86-ht9bn" podUID="21ba59b0-85ea-4547-be79-29c9a4008a3a" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.14:10250/livez\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:00.650904 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:39:00.617481    4559 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.386s"
Oct 29 22:39:00.650904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:00.617598    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"]
Oct 29 22:39:00.650904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:00.617903    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:39:00.650904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:00.617978    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:39:00.650904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:00.618037    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:39:00.650904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:00.618092    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:39:00.650904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:00.645837    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:01.043975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.035772    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:01.043975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.036513    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q" podUID="b258ed23-5da1-4feb-9b5f-a7de1f12d8b0" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 29 22:39:01.043975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.036611    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:39:01.099442 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.099387    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:39:01.358156 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.358044    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:01.407079 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.407044    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:39:01.484839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.437130    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 22:39:01.489849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.485919    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:39:01.489849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.486003    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t" podUID="62ebc209-264e-49db-87fd-ca9c789386b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:01.560312 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.560282    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:39:01.611044 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.594921    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:01.697837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.695962    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:01.733591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.733557    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:39:01.753527 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.753491    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-workload-socket\") pod \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") "
Oct 29 22:39:01.983983 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.754064    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:39:01.984333 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.862008    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t" event={"ID":"62ebc209-264e-49db-87fd-ca9c789386b4","Type":"ContainerDied","Data":"8eedd185a5e7a0ecfd2730cb0fb55be2896e837ba37b19eba7ecdcbbdb7aae61"}
Oct 29 22:39:01.984550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.984533    4559 scope.go:122] "RemoveContainer" containerID="c4fe2ada8cd77fbef664dac65577b1068589f03db9641ae83e3ab5adddfde0c1"
Oct 29 22:39:01.986952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.884154    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-workload-socket" pod "6a3bfdf3-5fca-44b9-88d7-bf525139ee93" (UID: "6a3bfdf3-5fca-44b9-88d7-bf525139ee93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:01.987184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.987158    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-workload-socket\") pod \"62ebc209-264e-49db-87fd-ca9c789386b4\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") "
Oct 29 22:39:01.987376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.987343    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62ebc209-264e-49db-87fd-ca9c789386b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ebc209-264e-49db-87fd-ca9c789386b4-istio-podinfo\") pod \"62ebc209-264e-49db-87fd-ca9c789386b4\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") "
Oct 29 22:39:01.990955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.990929    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-envoy\") pod \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") "
Oct 29 22:39:01.991196 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.991162    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ebc209-264e-49db-87fd-ca9c789386b4-kube-api-access-b5xc6\" (UniqueName: \"kubernetes.io/projected/62ebc209-264e-49db-87fd-ca9c789386b4-kube-api-access-b5xc6\") pod \"62ebc209-264e-49db-87fd-ca9c789386b4\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") "
Oct 29 22:39:01.997044 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.997016    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-kube-api-access-dd2kv\" (UniqueName: \"kubernetes.io/projected/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-kube-api-access-dd2kv\") pod \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") "
Oct 29 22:39:01.997323 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.997304    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-istio-data\") pod \"62ebc209-264e-49db-87fd-ca9c789386b4\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") "
Oct 29 22:39:01.997508 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.997482    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-podinfo\") pod \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") "
Oct 29 22:39:01.997664 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:01.997647    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-data\") pod \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") "
Oct 29 22:39:02.004970 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.004936    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-workload-socket" pod "62ebc209-264e-49db-87fd-ca9c789386b4" (UID: "62ebc209-264e-49db-87fd-ca9c789386b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:02.005567 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.005543    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-istio-envoy\") pod \"62ebc209-264e-49db-87fd-ca9c789386b4\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") "
Oct 29 22:39:02.005770 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.005753    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ebc209-264e-49db-87fd-ca9c789386b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ebc209-264e-49db-87fd-ca9c789386b4-istiod-ca-cert\") pod \"62ebc209-264e-49db-87fd-ca9c789386b4\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") "
Oct 29 22:39:02.006072 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.006040    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-credential-socket\") pod \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") "
Oct 29 22:39:02.006468 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.006452    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-credential-socket\") pod \"62ebc209-264e-49db-87fd-ca9c789386b4\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") "
Oct 29 22:39:02.011881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.007909    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-workload-certs\") pod \"62ebc209-264e-49db-87fd-ca9c789386b4\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") "
Oct 29 22:39:02.012159 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.012138    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-workload-certs\") pod \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") "
Oct 29 22:39:02.012457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.012440    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ebc209-264e-49db-87fd-ca9c789386b4-istio-token\" (UniqueName: \"kubernetes.io/projected/62ebc209-264e-49db-87fd-ca9c789386b4-istio-token\") pod \"62ebc209-264e-49db-87fd-ca9c789386b4\" (UID: \"62ebc209-264e-49db-87fd-ca9c789386b4\") "
Oct 29 22:39:02.032367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.032326    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-data" pod "6a3bfdf3-5fca-44b9-88d7-bf525139ee93" (UID: "6a3bfdf3-5fca-44b9-88d7-bf525139ee93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:02.131187 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.041976    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:02.633083 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.632996    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:02.633726 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.633703    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:02.634348 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.042209    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-credential-socket" pod "6a3bfdf3-5fca-44b9-88d7-bf525139ee93" (UID: "6a3bfdf3-5fca-44b9-88d7-bf525139ee93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:02.634475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.042428    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-credential-socket" pod "62ebc209-264e-49db-87fd-ca9c789386b4" (UID: "62ebc209-264e-49db-87fd-ca9c789386b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:02.634568 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.042616    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-workload-certs" pod "62ebc209-264e-49db-87fd-ca9c789386b4" (UID: "62ebc209-264e-49db-87fd-ca9c789386b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:02.634645 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.042792    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-workload-certs" pod "6a3bfdf3-5fca-44b9-88d7-bf525139ee93" (UID: "6a3bfdf3-5fca-44b9-88d7-bf525139ee93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:02.634723 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.073211    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ebc209-264e-49db-87fd-ca9c789386b4-istiod-ca-cert" pod "62ebc209-264e-49db-87fd-ca9c789386b4" (UID: "62ebc209-264e-49db-87fd-ca9c789386b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:39:02.634802 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.329629    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-istio-data" pod "62ebc209-264e-49db-87fd-ca9c789386b4" (UID: "62ebc209-264e-49db-87fd-ca9c789386b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:02.634905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.043875    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:02.660903 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.043178    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:02.685453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.316731    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:02.685453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.322220    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:39:02.685453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.329337    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:02.685453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.382063    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": context deadline exceeded"
Oct 29 22:39:02.685453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.603478    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:39:02.685453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.667838    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ebc209-264e-49db-87fd-ca9c789386b4-istio-token" pod "62ebc209-264e-49db-87fd-ca9c789386b4" (UID: "62ebc209-264e-49db-87fd-ca9c789386b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:02.685453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.671025    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62ebc209-264e-49db-87fd-ca9c789386b4-istio-podinfo" pod "62ebc209-264e-49db-87fd-ca9c789386b4" (UID: "62ebc209-264e-49db-87fd-ca9c789386b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:39:02.685453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.671115    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-istio-envoy" pod "62ebc209-264e-49db-87fd-ca9c789386b4" (UID: "62ebc209-264e-49db-87fd-ca9c789386b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:02.699877 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.698392    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:02.699877 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.698893    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-podinfo" pod "6a3bfdf3-5fca-44b9-88d7-bf525139ee93" (UID: "6a3bfdf3-5fca-44b9-88d7-bf525139ee93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:39:02.706481 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.705572    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ebc209-264e-49db-87fd-ca9c789386b4-kube-api-access-b5xc6" pod "62ebc209-264e-49db-87fd-ca9c789386b4" (UID: "62ebc209-264e-49db-87fd-ca9c789386b4"). InnerVolumeSpecName "kube-api-access-b5xc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:02.706481 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.706424    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:39:02.709888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.709860    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:39:02.761840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.745581    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-token\" (UniqueName: \"kubernetes.io/projected/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-token\") pod \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") "
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.798535    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-workload-certs\") pod \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") "
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.798802    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-data\") pod \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") "
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.798957    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-workload-socket\") pod \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") "
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.799063    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-credential-socket\") pod \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") "
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.799150    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-envoy\") pod \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") "
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.799319    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-kube-api-access-j4xxr\" (UniqueName: \"kubernetes.io/projected/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-kube-api-access-j4xxr\") pod \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") "
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.799350    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-podinfo\") pod \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") "
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.799421    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istiod-ca-cert\") pod \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\" (UID: \"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518\") "
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.799561    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istiod-ca-cert\") pod \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") "
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.799601    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-token\" (UniqueName: \"kubernetes.io/projected/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-token\") pod \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\" (UID: \"6a3bfdf3-5fca-44b9-88d7-bf525139ee93\") "
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.800996    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.801024    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.801096    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.801150    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.801165    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ebc209-264e-49db-87fd-ca9c789386b4-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.801213    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ebc209-264e-49db-87fd-ca9c789386b4-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.801228    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5xc6\" (UniqueName: \"kubernetes.io/projected/62ebc209-264e-49db-87fd-ca9c789386b4-kube-api-access-b5xc6\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.801247    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.801263    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.801276    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ebc209-264e-49db-87fd-ca9c789386b4-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.801290    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ebc209-264e-49db-87fd-ca9c789386b4-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.764996    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-envoy" pod "6a3bfdf3-5fca-44b9-88d7-bf525139ee93" (UID: "6a3bfdf3-5fca-44b9-88d7-bf525139ee93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.811994    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-workload-certs" pod "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" (UID: "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:02.823264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.798480    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:39:02.849832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.849586    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-data" pod "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" (UID: "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:02.874107 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.874059    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-kube-api-access-dd2kv" pod "6a3bfdf3-5fca-44b9-88d7-bf525139ee93" (UID: "6a3bfdf3-5fca-44b9-88d7-bf525139ee93"). InnerVolumeSpecName "kube-api-access-dd2kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:02.920319 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.920271    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-workload-socket" pod "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" (UID: "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:02.942850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.942793    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istiod-ca-cert" pod "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" (UID: "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:39:02.977956 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.957092    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-credential-socket" pod "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" (UID: "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:02.991917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:02.991537    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istiod-ca-cert" pod "6a3bfdf3-5fca-44b9-88d7-bf525139ee93" (UID: "6a3bfdf3-5fca-44b9-88d7-bf525139ee93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:39:03.026219 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.026186    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:03.026468 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.026455    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:03.026558 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.026548    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:03.026638 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.026628    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd2kv\" (UniqueName: \"kubernetes.io/projected/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-kube-api-access-dd2kv\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:03.026715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.026701    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:03.026798 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.026789    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:03.026914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.026900    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:03.026989 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.026980    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:03.117492 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.117451    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:39:03.124260 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.124229    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:39:03.161389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.135931    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-podinfo" pod "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" (UID: "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:39:03.161389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.149006    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-envoy" pod "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" (UID: "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:03.189512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.189480    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:03.194917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.194886    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:03.200902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.197856    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-token" pod "6a3bfdf3-5fca-44b9-88d7-bf525139ee93" (UID: "6a3bfdf3-5fca-44b9-88d7-bf525139ee93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:03.202130 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.202102    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"
Oct 29 22:39:03.203026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.202989    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82" event={"ID":"6a3bfdf3-5fca-44b9-88d7-bf525139ee93","Type":"ContainerDied","Data":"c7c34923d15368d754f5bfa8c08a60505348a698d55a31275d74282c27b225d5"}
Oct 29 22:39:03.203144 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.203059    4559 scope.go:122] "RemoveContainer" containerID="1c153840f8535885d6955996c97c2fc506a2450fff700e6c8ed7ecf1be7fc0e8"
Oct 29 22:39:03.203144 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.203097    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-token" pod "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" (UID: "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:03.207449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.207401    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:39:03.217881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.216758    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-kube-api-access-j4xxr" pod "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" (UID: "4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518"). InnerVolumeSpecName "kube-api-access-j4xxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:03.222693 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.222668    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"
Oct 29 22:39:03.321921 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.321889    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4xxr\" (UniqueName: \"kubernetes.io/projected/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-kube-api-access-j4xxr\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:03.322758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.322712    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv" event={"ID":"4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518","Type":"ContainerDied","Data":"0da01749f46e18ccfc275b9cc77a4484c3166d4325ea47b0581d1584552b3f90"}
Oct 29 22:39:03.322901 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.322877    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:39:03.323493 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.323246    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a3bfdf3-5fca-44b9-88d7-bf525139ee93-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:03.324960 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.324932    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:03.325078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.323896    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:39:03.326063 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.326040    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"
Oct 29 22:39:03.355912 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.344781    4559 scope.go:122] "RemoveContainer" containerID="b908886771c48e6baeeb8200680778da8bec333b878c5ffd52ffac00bfb9532a"
Oct 29 22:39:03.554161 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.554073    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"]
Oct 29 22:39:03.558846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.558779    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-8656t"]
Oct 29 22:39:03.594098 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.594067    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"]
Oct 29 22:39:03.600824 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.600781    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-7glgv"]
Oct 29 22:39:03.674610 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.674566    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"]
Oct 29 22:39:03.683697 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:03.683660    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-ntt82"]
Oct 29 22:39:04.231038 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:04.230990    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59" podUID="7cb532d0-c0f2-4a78-94ce-5b6272b175e5" containerName="payload-processing" containerID="cri-o://cf43416fec668be01b6c184a76063f61b0fcce335935d4372d52fdb85d45481b" gracePeriod=25
Oct 29 22:39:04.632401 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:04.632369    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"
Oct 29 22:39:04.661928 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:04.661889    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb532d0-c0f2-4a78-94ce-5b6272b175e5-kube-api-access-9rnd8\" (UniqueName: \"kubernetes.io/projected/7cb532d0-c0f2-4a78-94ce-5b6272b175e5-kube-api-access-9rnd8\") pod \"7cb532d0-c0f2-4a78-94ce-5b6272b175e5\" (UID: \"7cb532d0-c0f2-4a78-94ce-5b6272b175e5\") "
Oct 29 22:39:04.662132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:04.661984    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb532d0-c0f2-4a78-94ce-5b6272b175e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb532d0-c0f2-4a78-94ce-5b6272b175e5-plugins-config-volume\") pod \"7cb532d0-c0f2-4a78-94ce-5b6272b175e5\" (UID: \"7cb532d0-c0f2-4a78-94ce-5b6272b175e5\") "
Oct 29 22:39:04.662540 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:04.662507    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb532d0-c0f2-4a78-94ce-5b6272b175e5-plugins-config-volume" pod "7cb532d0-c0f2-4a78-94ce-5b6272b175e5" (UID: "7cb532d0-c0f2-4a78-94ce-5b6272b175e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:39:04.663458 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:04.663431    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb532d0-c0f2-4a78-94ce-5b6272b175e5-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:04.668183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:04.667967    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb532d0-c0f2-4a78-94ce-5b6272b175e5-kube-api-access-9rnd8" pod "7cb532d0-c0f2-4a78-94ce-5b6272b175e5" (UID: "7cb532d0-c0f2-4a78-94ce-5b6272b175e5"). InnerVolumeSpecName "kube-api-access-9rnd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:04.765137 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:04.765059    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rnd8\" (UniqueName: \"kubernetes.io/projected/7cb532d0-c0f2-4a78-94ce-5b6272b175e5-kube-api-access-9rnd8\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:05.233049 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.233012    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" path="/var/lib/kubelet/pods/4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518/volumes"
Oct 29 22:39:05.233600 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.233583    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ebc209-264e-49db-87fd-ca9c789386b4" path="/var/lib/kubelet/pods/62ebc209-264e-49db-87fd-ca9c789386b4/volumes"
Oct 29 22:39:05.234072 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.234056    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a3bfdf3-5fca-44b9-88d7-bf525139ee93" path="/var/lib/kubelet/pods/6a3bfdf3-5fca-44b9-88d7-bf525139ee93/volumes"
Oct 29 22:39:05.311109 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.311070    4559 generic.go:356] "Generic (PLEG): container finished" podID="7cb532d0-c0f2-4a78-94ce-5b6272b175e5" containerID="cf43416fec668be01b6c184a76063f61b0fcce335935d4372d52fdb85d45481b" exitCode=0
Oct 29 22:39:05.311109 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.311117    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"
Oct 29 22:39:05.311358 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.311160    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59" event={"ID":"7cb532d0-c0f2-4a78-94ce-5b6272b175e5","Type":"ContainerDied","Data":"cf43416fec668be01b6c184a76063f61b0fcce335935d4372d52fdb85d45481b"}
Oct 29 22:39:05.311358 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.311214    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59" event={"ID":"7cb532d0-c0f2-4a78-94ce-5b6272b175e5","Type":"ContainerDied","Data":"aa3ca0a9c23c1f80f44f01d29b4a17a12dd72d1b744cd9d17ba7f3518dda97e4"}
Oct 29 22:39:05.311358 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.311240    4559 scope.go:122] "RemoveContainer" containerID="cf43416fec668be01b6c184a76063f61b0fcce335935d4372d52fdb85d45481b"
Oct 29 22:39:05.323862 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.323803    4559 scope.go:122] "RemoveContainer" containerID="cf43416fec668be01b6c184a76063f61b0fcce335935d4372d52fdb85d45481b"
Oct 29 22:39:05.324227 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:39:05.324204    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf43416fec668be01b6c184a76063f61b0fcce335935d4372d52fdb85d45481b\": container with ID starting with cf43416fec668be01b6c184a76063f61b0fcce335935d4372d52fdb85d45481b not found: ID does not exist" containerID="cf43416fec668be01b6c184a76063f61b0fcce335935d4372d52fdb85d45481b"
Oct 29 22:39:05.324288 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.324238    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf43416fec668be01b6c184a76063f61b0fcce335935d4372d52fdb85d45481b"} err="failed to get container status \"cf43416fec668be01b6c184a76063f61b0fcce335935d4372d52fdb85d45481b\": rpc error: code = NotFound desc = could not find container \"cf43416fec668be01b6c184a76063f61b0fcce335935d4372d52fdb85d45481b\": container with ID starting with cf43416fec668be01b6c184a76063f61b0fcce335935d4372d52fdb85d45481b not found: ID does not exist"
Oct 29 22:39:05.366683 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.366643    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"]
Oct 29 22:39:05.381724 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.381683    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cae55b-9587db5fb-8nx59"]
Oct 29 22:39:05.620043 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.619949    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2" podUID="8f5cb439-a622-48d5-9b98-b76c8b94dd85" containerName="payload-pre-processing" containerID="cri-o://3476f7edcf2c0be6eb10b7aa1370bcd284b33c96724e227e0c17495f6c7496d2" gracePeriod=25
Oct 29 22:39:05.950752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.950711    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"
Oct 29 22:39:05.987265 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.987218    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5cb439-a622-48d5-9b98-b76c8b94dd85-kube-api-access-zl4mz\" (UniqueName: \"kubernetes.io/projected/8f5cb439-a622-48d5-9b98-b76c8b94dd85-kube-api-access-zl4mz\") pod \"8f5cb439-a622-48d5-9b98-b76c8b94dd85\" (UID: \"8f5cb439-a622-48d5-9b98-b76c8b94dd85\") "
Oct 29 22:39:05.987693 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.987668    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f5cb439-a622-48d5-9b98-b76c8b94dd85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f5cb439-a622-48d5-9b98-b76c8b94dd85-plugins-config-volume\") pod \"8f5cb439-a622-48d5-9b98-b76c8b94dd85\" (UID: \"8f5cb439-a622-48d5-9b98-b76c8b94dd85\") "
Oct 29 22:39:05.988094 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.988069    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f5cb439-a622-48d5-9b98-b76c8b94dd85-plugins-config-volume" pod "8f5cb439-a622-48d5-9b98-b76c8b94dd85" (UID: "8f5cb439-a622-48d5-9b98-b76c8b94dd85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:39:05.988302 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.988282    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f5cb439-a622-48d5-9b98-b76c8b94dd85-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:05.989551 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:05.989528    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5cb439-a622-48d5-9b98-b76c8b94dd85-kube-api-access-zl4mz" pod "8f5cb439-a622-48d5-9b98-b76c8b94dd85" (UID: "8f5cb439-a622-48d5-9b98-b76c8b94dd85"). InnerVolumeSpecName "kube-api-access-zl4mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:06.096835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:06.090625    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl4mz\" (UniqueName: \"kubernetes.io/projected/8f5cb439-a622-48d5-9b98-b76c8b94dd85-kube-api-access-zl4mz\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:06.321373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:06.321286    4559 generic.go:356] "Generic (PLEG): container finished" podID="8f5cb439-a622-48d5-9b98-b76c8b94dd85" containerID="3476f7edcf2c0be6eb10b7aa1370bcd284b33c96724e227e0c17495f6c7496d2" exitCode=0
Oct 29 22:39:06.321373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:06.321344    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"
Oct 29 22:39:06.321618 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:06.321392    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2" event={"ID":"8f5cb439-a622-48d5-9b98-b76c8b94dd85","Type":"ContainerDied","Data":"3476f7edcf2c0be6eb10b7aa1370bcd284b33c96724e227e0c17495f6c7496d2"}
Oct 29 22:39:06.321618 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:06.321458    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2" event={"ID":"8f5cb439-a622-48d5-9b98-b76c8b94dd85","Type":"ContainerDied","Data":"9ee242ba187652d94f46d64f55c6933dbba18b0ca3b3e69a8094e5f86b0cdfe6"}
Oct 29 22:39:06.321618 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:06.321493    4559 scope.go:122] "RemoveContainer" containerID="3476f7edcf2c0be6eb10b7aa1370bcd284b33c96724e227e0c17495f6c7496d2"
Oct 29 22:39:06.334411 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:06.334382    4559 scope.go:122] "RemoveContainer" containerID="3476f7edcf2c0be6eb10b7aa1370bcd284b33c96724e227e0c17495f6c7496d2"
Oct 29 22:39:06.334754 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:39:06.334725    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3476f7edcf2c0be6eb10b7aa1370bcd284b33c96724e227e0c17495f6c7496d2\": container with ID starting with 3476f7edcf2c0be6eb10b7aa1370bcd284b33c96724e227e0c17495f6c7496d2 not found: ID does not exist" containerID="3476f7edcf2c0be6eb10b7aa1370bcd284b33c96724e227e0c17495f6c7496d2"
Oct 29 22:39:06.334916 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:06.334766    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3476f7edcf2c0be6eb10b7aa1370bcd284b33c96724e227e0c17495f6c7496d2"} err="failed to get container status \"3476f7edcf2c0be6eb10b7aa1370bcd284b33c96724e227e0c17495f6c7496d2\": rpc error: code = NotFound desc = could not find container \"3476f7edcf2c0be6eb10b7aa1370bcd284b33c96724e227e0c17495f6c7496d2\": container with ID starting with 3476f7edcf2c0be6eb10b7aa1370bcd284b33c96724e227e0c17495f6c7496d2 not found: ID does not exist"
Oct 29 22:39:06.376738 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:06.376702    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"]
Oct 29 22:39:06.388641 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:06.388608    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cae55b-84d5658d5f-gzwp2"]
Oct 29 22:39:07.233255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:07.233209    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb532d0-c0f2-4a78-94ce-5b6272b175e5" path="/var/lib/kubelet/pods/7cb532d0-c0f2-4a78-94ce-5b6272b175e5/volumes"
Oct 29 22:39:07.233930 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:07.233908    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f5cb439-a622-48d5-9b98-b76c8b94dd85" path="/var/lib/kubelet/pods/8f5cb439-a622-48d5-9b98-b76c8b94dd85/volumes"
Oct 29 22:39:08.314834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.255708    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"]
Oct 29 22:39:08.314834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.257207    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6a3bfdf3-5fca-44b9-88d7-bf525139ee93" containerName="istio-proxy"
Oct 29 22:39:08.314834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.257231    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3bfdf3-5fca-44b9-88d7-bf525139ee93" containerName="istio-proxy"
Oct 29 22:39:08.314834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.257259    4559 cpu_manager.go:414] "containerMap: removing container" podUID="8f5cb439-a622-48d5-9b98-b76c8b94dd85" containerName="payload-pre-processing"
Oct 29 22:39:08.314834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.257267    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5cb439-a622-48d5-9b98-b76c8b94dd85" containerName="payload-pre-processing"
Oct 29 22:39:08.314834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.257293    4559 cpu_manager.go:414] "containerMap: removing container" podUID="62ebc209-264e-49db-87fd-ca9c789386b4" containerName="istio-proxy"
Oct 29 22:39:08.314834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.257301    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ebc209-264e-49db-87fd-ca9c789386b4" containerName="istio-proxy"
Oct 29 22:39:08.314834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.257351    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" containerName="istio-proxy"
Oct 29 22:39:08.314834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.257359    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" containerName="istio-proxy"
Oct 29 22:39:08.314834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.257377    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7cb532d0-c0f2-4a78-94ce-5b6272b175e5" containerName="payload-processing"
Oct 29 22:39:08.314834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.257385    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb532d0-c0f2-4a78-94ce-5b6272b175e5" containerName="payload-processing"
Oct 29 22:39:08.314834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.257564    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a3bfdf3-5fca-44b9-88d7-bf525139ee93" containerName="istio-proxy"
Oct 29 22:39:08.314834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.257579    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5cb439-a622-48d5-9b98-b76c8b94dd85" containerName="payload-pre-processing"
Oct 29 22:39:08.314834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.257594    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ebc209-264e-49db-87fd-ca9c789386b4" containerName="istio-proxy"
Oct 29 22:39:08.314834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.257608    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aec0aeb-95b6-4ea8-9e61-c0a97e6b1518" containerName="istio-proxy"
Oct 29 22:39:08.314834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.257617    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb532d0-c0f2-4a78-94ce-5b6272b175e5" containerName="payload-processing"
Oct 29 22:39:08.347837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.347247    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:39:08.362835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.352977    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"]
Oct 29 22:39:08.396112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.392468    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-metrics-certs\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j\" (UID: \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:39:08.396112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.392700    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlt88\" (UniqueName: \"kubernetes.io/projected/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-kube-api-access-tlt88\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j\" (UID: \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:39:08.396112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.392750    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-maas-api-tls\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j\" (UID: \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:39:08.396112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.392776    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-openshift-service-ca\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j\" (UID: \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:39:08.396666 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.396598    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1fa4b8b6-serving-cert\""
Oct 29 22:39:08.494116 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.494080    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-metrics-certs\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j\" (UID: \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:39:08.496425 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.494271    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlt88\" (UniqueName: \"kubernetes.io/projected/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-kube-api-access-tlt88\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j\" (UID: \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:39:08.496425 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.494305    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-maas-api-tls\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j\" (UID: \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:39:08.496425 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.494321    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-openshift-service-ca\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j\" (UID: \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:39:08.496425 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.494951    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-openshift-service-ca\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j\" (UID: \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:39:08.497272 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.497249    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-maas-api-tls\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j\" (UID: \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:39:08.497582 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.497558    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-metrics-certs\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j\" (UID: \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:39:08.513788 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.513759    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlt88\" (UniqueName: \"kubernetes.io/projected/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-kube-api-access-tlt88\") pod \"maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j\" (UID: \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:39:08.723121 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:08.723040    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:39:09.045287 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:09.045254    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"]
Oct 29 22:39:09.047609 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:39:09.047577    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4e3a3a_c3fa_4dcd_aa66_f5d7158bfedb.slice/crio-8c50695992c7a7dd1e11ed60470496d2489689625d6e6cf040311198be687eed WatchSource:0}: Error finding container 8c50695992c7a7dd1e11ed60470496d2489689625d6e6cf040311198be687eed: Status 404 returned error can't find the container with id 8c50695992c7a7dd1e11ed60470496d2489689625d6e6cf040311198be687eed
Oct 29 22:39:09.477880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:09.461679    4559 generic.go:356] "Generic (PLEG): container finished" podID="5d1bb1ea-189a-4033-b702-7fba8db86cdf" containerID="7ffc1e613e0ea35e105c2caa1119cf09feeebafd0223e8a9b3ed0582f498cb3d" exitCode=0
Oct 29 22:39:09.477880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:09.461881    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv" event={"ID":"5d1bb1ea-189a-4033-b702-7fba8db86cdf","Type":"ContainerDied","Data":"7ffc1e613e0ea35e105c2caa1119cf09feeebafd0223e8a9b3ed0582f498cb3d"}
Oct 29 22:39:09.477880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:09.462451    4559 scope.go:122] "RemoveContainer" containerID="7ffc1e613e0ea35e105c2caa1119cf09feeebafd0223e8a9b3ed0582f498cb3d"
Oct 29 22:39:09.503368 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:09.480719    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:39:09.503368 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:09.480857    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:39:09.503368 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:09.480907    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:39:09.505763 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:09.505678    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j" event={"ID":"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb","Type":"ContainerStarted","Data":"8c50695992c7a7dd1e11ed60470496d2489689625d6e6cf040311198be687eed"}
Oct 29 22:39:10.515338 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:10.515301    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j" event={"ID":"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb","Type":"ContainerStarted","Data":"a4b0aa60b86b3b3444aa5a7120b6665765c7c18c75bc2e5abec677a24fc75be4"}
Oct 29 22:39:10.515852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:10.515392    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:39:10.517686 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:10.517661    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv" event={"ID":"5d1bb1ea-189a-4033-b702-7fba8db86cdf","Type":"ContainerStarted","Data":"f0ebdfc5b88f77117f530d3be3ced813f1cebaf535b2b20f5031be77d46bc2b3"}
Oct 29 22:39:10.562940 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:10.562787    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j" podStartSLOduration=3.197306892 podStartE2EDuration="3.562766785s" totalImagesPullingTime="365.459893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:10.556932018 +0000 UTC m=+1489.468375571" watchObservedRunningTime="2026-10-29 22:39:10.562766785 +0000 UTC m=+1489.474210340"
Oct 29 22:39:11.421758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.421723    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:39:11.422947 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.422920    4559 patch_prober.go:29] interesting pod/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:39:11.423067 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.422971    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv" podUID="5d1bb1ea-189a-4033-b702-7fba8db86cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 29 22:39:11.601537 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.601499    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"]
Oct 29 22:39:11.606923 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.606898    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:39:11.625464 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.625423    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"]
Oct 29 22:39:11.654488 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.654425    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c1750b-serving-cert\""
Oct 29 22:39:11.694910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.690782    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2zb8\" (UniqueName: \"kubernetes.io/projected/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-kube-api-access-z2zb8\") pod \"maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8\" (UID: \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:39:11.694910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.690870    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-metrics-certs\") pod \"maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8\" (UID: \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:39:11.694910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.691166    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8\" (UID: \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:39:11.694910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.691263    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-maas-api-tls\") pod \"maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8\" (UID: \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:39:11.793150 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.793099    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2zb8\" (UniqueName: \"kubernetes.io/projected/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-kube-api-access-z2zb8\") pod \"maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8\" (UID: \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:39:11.793329 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.793177    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-metrics-certs\") pod \"maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8\" (UID: \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:39:11.793375 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.793337    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8\" (UID: \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:39:11.793419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.793407    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-maas-api-tls\") pod \"maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8\" (UID: \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:39:11.794035 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.793957    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8\" (UID: \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:39:11.796019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.795993    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-metrics-certs\") pod \"maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8\" (UID: \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:39:11.796123 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.795997    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-maas-api-tls\") pod \"maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8\" (UID: \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:39:11.813035 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.812998    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2zb8\" (UniqueName: \"kubernetes.io/projected/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-kube-api-access-z2zb8\") pod \"maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8\" (UID: \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:39:11.929393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:11.929095    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:39:12.428840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:12.427255    4559 patch_prober.go:29] interesting pod/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:39:12.428840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:12.427329    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv" podUID="5d1bb1ea-189a-4033-b702-7fba8db86cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 29 22:39:12.453595 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:12.435025    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:39:12.512098 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:39:12.512056    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7e1e83_bb79_4e63_9c0d_4c9112deb2e0.slice/crio-ac9629d14fc697744213de61f74268cfbdcaebe46cf92df8648a4dbb1ac02c4f WatchSource:0}: Error finding container ac9629d14fc697744213de61f74268cfbdcaebe46cf92df8648a4dbb1ac02c4f: Status 404 returned error can't find the container with id ac9629d14fc697744213de61f74268cfbdcaebe46cf92df8648a4dbb1ac02c4f
Oct 29 22:39:12.516251 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:12.516221    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"]
Oct 29 22:39:12.577093 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:12.577053    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8" event={"ID":"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0","Type":"ContainerStarted","Data":"ac9629d14fc697744213de61f74268cfbdcaebe46cf92df8648a4dbb1ac02c4f"}
Oct 29 22:39:13.421515 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:13.421421    4559 patch_prober.go:29] interesting pod/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:39:13.421515 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:13.421485    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv" podUID="5d1bb1ea-189a-4033-b702-7fba8db86cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 29 22:39:13.591318 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:13.591280    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8" event={"ID":"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0","Type":"ContainerStarted","Data":"3ce2624f1befcf83f9c750a74ddd4a04c4dcfba502fc8d63a96459e8f947badc"}
Oct 29 22:39:13.592075 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:13.592054    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:39:13.712257 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:39:13.712164    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:39:14.264836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:14.263792    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8" podStartSLOduration=2.931266869 podStartE2EDuration="3.263768326s" totalImagesPullingTime="332.501457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:13.649073258 +0000 UTC m=+1492.560516812" watchObservedRunningTime="2026-10-29 22:39:14.263768326 +0000 UTC m=+1493.175211881"
Oct 29 22:39:14.265078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:14.264900    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"]
Oct 29 22:39:14.286486 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:14.286453    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"]
Oct 29 22:39:14.422489 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:14.422448    4559 patch_prober.go:29] interesting pod/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:39:14.423204 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:14.423136    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv" podUID="5d1bb1ea-189a-4033-b702-7fba8db86cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 29 22:39:15.428674 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:15.428220    4559 patch_prober.go:29] interesting pod/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:39:15.428674 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:15.428296    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv" podUID="5d1bb1ea-189a-4033-b702-7fba8db86cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 29 22:39:16.423395 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:16.423354    4559 patch_prober.go:29] interesting pod/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:39:16.423627 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:16.423435    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv" podUID="5d1bb1ea-189a-4033-b702-7fba8db86cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 29 22:39:16.586634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:16.586592    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:39:17.421421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:17.421385    4559 patch_prober.go:29] interesting pod/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:39:17.421632 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:17.421448    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv" podUID="5d1bb1ea-189a-4033-b702-7fba8db86cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 29 22:39:18.426977 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:18.426945    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:39:18.428122 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:18.428101    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:39:19.267661 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.267613    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6" podUID="63a99d09-6876-4624-99f0-3fb56f3b3935" containerName="payload-processing" containerID="cri-o://057f7a5667864e004ebe1c0e2867ef8c7845e90a34340aff87fa03f961f94fef" gracePeriod=25
Oct 29 22:39:19.287224 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.287181    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj" podUID="89e9567e-9768-4800-8fee-cbe5b93d7850" containerName="payload-pre-processing" containerID="cri-o://6dbefe001dca9fd07ddd36e9216c8339ff22890dc8e154a53115f57b058ad0e9" gracePeriod=25
Oct 29 22:39:19.603072 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.603038    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"
Oct 29 22:39:19.660105 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.660079    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"
Oct 29 22:39:19.710479 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.710449    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:39:19.712538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.712506    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a99d09-6876-4624-99f0-3fb56f3b3935-kube-api-access-rb2cc\" (UniqueName: \"kubernetes.io/projected/63a99d09-6876-4624-99f0-3fb56f3b3935-kube-api-access-rb2cc\") pod \"63a99d09-6876-4624-99f0-3fb56f3b3935\" (UID: \"63a99d09-6876-4624-99f0-3fb56f3b3935\") "
Oct 29 22:39:19.712803 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.712781    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63a99d09-6876-4624-99f0-3fb56f3b3935-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a99d09-6876-4624-99f0-3fb56f3b3935-plugins-config-volume\") pod \"63a99d09-6876-4624-99f0-3fb56f3b3935\" (UID: \"63a99d09-6876-4624-99f0-3fb56f3b3935\") "
Oct 29 22:39:19.713194 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.713169    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a99d09-6876-4624-99f0-3fb56f3b3935-plugins-config-volume" pod "63a99d09-6876-4624-99f0-3fb56f3b3935" (UID: "63a99d09-6876-4624-99f0-3fb56f3b3935"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:39:19.713708 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.713682    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a99d09-6876-4624-99f0-3fb56f3b3935-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:19.714995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.714967    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a99d09-6876-4624-99f0-3fb56f3b3935-kube-api-access-rb2cc" pod "63a99d09-6876-4624-99f0-3fb56f3b3935" (UID: "63a99d09-6876-4624-99f0-3fb56f3b3935"). InnerVolumeSpecName "kube-api-access-rb2cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:19.814606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.814504    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e9567e-9768-4800-8fee-cbe5b93d7850-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e9567e-9768-4800-8fee-cbe5b93d7850-plugins-config-volume\") pod \"89e9567e-9768-4800-8fee-cbe5b93d7850\" (UID: \"89e9567e-9768-4800-8fee-cbe5b93d7850\") "
Oct 29 22:39:19.814783 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.814726    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e9567e-9768-4800-8fee-cbe5b93d7850-kube-api-access-f4nnd\" (UniqueName: \"kubernetes.io/projected/89e9567e-9768-4800-8fee-cbe5b93d7850-kube-api-access-f4nnd\") pod \"89e9567e-9768-4800-8fee-cbe5b93d7850\" (UID: \"89e9567e-9768-4800-8fee-cbe5b93d7850\") "
Oct 29 22:39:19.814976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.814951    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e9567e-9768-4800-8fee-cbe5b93d7850-plugins-config-volume" pod "89e9567e-9768-4800-8fee-cbe5b93d7850" (UID: "89e9567e-9768-4800-8fee-cbe5b93d7850"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:39:19.817068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.817042    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e9567e-9768-4800-8fee-cbe5b93d7850-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:19.817240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.817214    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb2cc\" (UniqueName: \"kubernetes.io/projected/63a99d09-6876-4624-99f0-3fb56f3b3935-kube-api-access-rb2cc\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:19.817644 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.817621    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e9567e-9768-4800-8fee-cbe5b93d7850-kube-api-access-f4nnd" pod "89e9567e-9768-4800-8fee-cbe5b93d7850" (UID: "89e9567e-9768-4800-8fee-cbe5b93d7850"). InnerVolumeSpecName "kube-api-access-f4nnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:19.891594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.891558    4559 generic.go:356] "Generic (PLEG): container finished" podID="63a99d09-6876-4624-99f0-3fb56f3b3935" containerID="057f7a5667864e004ebe1c0e2867ef8c7845e90a34340aff87fa03f961f94fef" exitCode=0
Oct 29 22:39:19.891786 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.891608    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"
Oct 29 22:39:19.891786 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.891638    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6" event={"ID":"63a99d09-6876-4624-99f0-3fb56f3b3935","Type":"ContainerDied","Data":"057f7a5667864e004ebe1c0e2867ef8c7845e90a34340aff87fa03f961f94fef"}
Oct 29 22:39:19.891786 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.891686    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6" event={"ID":"63a99d09-6876-4624-99f0-3fb56f3b3935","Type":"ContainerDied","Data":"5192e775069da513e90a847626d3b37d9c17acaf4dec86153c1a617af6a5a1e1"}
Oct 29 22:39:19.891786 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.891715    4559 scope.go:122] "RemoveContainer" containerID="057f7a5667864e004ebe1c0e2867ef8c7845e90a34340aff87fa03f961f94fef"
Oct 29 22:39:19.893188 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.893163    4559 generic.go:356] "Generic (PLEG): container finished" podID="89e9567e-9768-4800-8fee-cbe5b93d7850" containerID="6dbefe001dca9fd07ddd36e9216c8339ff22890dc8e154a53115f57b058ad0e9" exitCode=0
Oct 29 22:39:19.893300 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.893219    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"
Oct 29 22:39:19.893300 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.893251    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj" event={"ID":"89e9567e-9768-4800-8fee-cbe5b93d7850","Type":"ContainerDied","Data":"6dbefe001dca9fd07ddd36e9216c8339ff22890dc8e154a53115f57b058ad0e9"}
Oct 29 22:39:19.893401 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.893299    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj" event={"ID":"89e9567e-9768-4800-8fee-cbe5b93d7850","Type":"ContainerDied","Data":"972bb4093f788fe5e85be6c1891aff272fbbe301419550d6ae0d6592bdfa88e6"}
Oct 29 22:39:19.903458 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.903436    4559 scope.go:122] "RemoveContainer" containerID="057f7a5667864e004ebe1c0e2867ef8c7845e90a34340aff87fa03f961f94fef"
Oct 29 22:39:19.903782 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:39:19.903761    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057f7a5667864e004ebe1c0e2867ef8c7845e90a34340aff87fa03f961f94fef\": container with ID starting with 057f7a5667864e004ebe1c0e2867ef8c7845e90a34340aff87fa03f961f94fef not found: ID does not exist" containerID="057f7a5667864e004ebe1c0e2867ef8c7845e90a34340aff87fa03f961f94fef"
Oct 29 22:39:19.903873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.903795    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057f7a5667864e004ebe1c0e2867ef8c7845e90a34340aff87fa03f961f94fef"} err="failed to get container status \"057f7a5667864e004ebe1c0e2867ef8c7845e90a34340aff87fa03f961f94fef\": rpc error: code = NotFound desc = could not find container \"057f7a5667864e004ebe1c0e2867ef8c7845e90a34340aff87fa03f961f94fef\": container with ID starting with 057f7a5667864e004ebe1c0e2867ef8c7845e90a34340aff87fa03f961f94fef not found: ID does not exist"
Oct 29 22:39:19.903873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.903832    4559 scope.go:122] "RemoveContainer" containerID="6dbefe001dca9fd07ddd36e9216c8339ff22890dc8e154a53115f57b058ad0e9"
Oct 29 22:39:19.913934 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.913914    4559 scope.go:122] "RemoveContainer" containerID="6dbefe001dca9fd07ddd36e9216c8339ff22890dc8e154a53115f57b058ad0e9"
Oct 29 22:39:19.914210 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:39:19.914189    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dbefe001dca9fd07ddd36e9216c8339ff22890dc8e154a53115f57b058ad0e9\": container with ID starting with 6dbefe001dca9fd07ddd36e9216c8339ff22890dc8e154a53115f57b058ad0e9 not found: ID does not exist" containerID="6dbefe001dca9fd07ddd36e9216c8339ff22890dc8e154a53115f57b058ad0e9"
Oct 29 22:39:19.914259 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.914219    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dbefe001dca9fd07ddd36e9216c8339ff22890dc8e154a53115f57b058ad0e9"} err="failed to get container status \"6dbefe001dca9fd07ddd36e9216c8339ff22890dc8e154a53115f57b058ad0e9\": rpc error: code = NotFound desc = could not find container \"6dbefe001dca9fd07ddd36e9216c8339ff22890dc8e154a53115f57b058ad0e9\": container with ID starting with 6dbefe001dca9fd07ddd36e9216c8339ff22890dc8e154a53115f57b058ad0e9 not found: ID does not exist"
Oct 29 22:39:19.918889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.918864    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4nnd\" (UniqueName: \"kubernetes.io/projected/89e9567e-9768-4800-8fee-cbe5b93d7850-kube-api-access-f4nnd\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:19.947601 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.947560    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"]
Oct 29 22:39:19.951882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.951851    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-616da7-7747fb4947-5lbcj"]
Oct 29 22:39:19.983928 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.983891    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"]
Oct 29 22:39:19.990021 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:19.989989    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-616da7-8485684f4d-b7ww6"]
Oct 29 22:39:20.388206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:20.388169    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:39:21.231572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:21.231542    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63a99d09-6876-4624-99f0-3fb56f3b3935" path="/var/lib/kubelet/pods/63a99d09-6876-4624-99f0-3fb56f3b3935/volumes"
Oct 29 22:39:21.232118 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:21.232103    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e9567e-9768-4800-8fee-cbe5b93d7850" path="/var/lib/kubelet/pods/89e9567e-9768-4800-8fee-cbe5b93d7850/volumes"
Oct 29 22:39:21.465724 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:21.465690    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:39:21.473317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:21.473294    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:39:21.501305 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:21.501220    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"]
Oct 29 22:39:26.502127 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:26.502040    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn" podUID="2dd88602-acf5-41be-89f4-e5aa49e2a357" containerName="payload-processing" containerID="cri-o://3c229ca553c1469080f7d278e9951ba06002e1f778b7c272ed657a45474971f5" gracePeriod=25
Oct 29 22:39:26.871028 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:26.870998    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"
Oct 29 22:39:26.941853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:26.938549    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd88602-acf5-41be-89f4-e5aa49e2a357-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd88602-acf5-41be-89f4-e5aa49e2a357-plugins-config-volume\") pod \"2dd88602-acf5-41be-89f4-e5aa49e2a357\" (UID: \"2dd88602-acf5-41be-89f4-e5aa49e2a357\") "
Oct 29 22:39:26.941853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:26.938694    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd88602-acf5-41be-89f4-e5aa49e2a357-kube-api-access-jvcl8\" (UniqueName: \"kubernetes.io/projected/2dd88602-acf5-41be-89f4-e5aa49e2a357-kube-api-access-jvcl8\") pod \"2dd88602-acf5-41be-89f4-e5aa49e2a357\" (UID: \"2dd88602-acf5-41be-89f4-e5aa49e2a357\") "
Oct 29 22:39:26.943390 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:26.943350    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd88602-acf5-41be-89f4-e5aa49e2a357-plugins-config-volume" pod "2dd88602-acf5-41be-89f4-e5aa49e2a357" (UID: "2dd88602-acf5-41be-89f4-e5aa49e2a357"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:39:26.963062 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:26.963023    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd88602-acf5-41be-89f4-e5aa49e2a357-kube-api-access-jvcl8" pod "2dd88602-acf5-41be-89f4-e5aa49e2a357" (UID: "2dd88602-acf5-41be-89f4-e5aa49e2a357"). InnerVolumeSpecName "kube-api-access-jvcl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:27.059393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:27.059288    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd88602-acf5-41be-89f4-e5aa49e2a357-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:27.059393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:27.059326    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvcl8\" (UniqueName: \"kubernetes.io/projected/2dd88602-acf5-41be-89f4-e5aa49e2a357-kube-api-access-jvcl8\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:27.230067 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:27.230031    4559 generic.go:356] "Generic (PLEG): container finished" podID="2dd88602-acf5-41be-89f4-e5aa49e2a357" containerID="3c229ca553c1469080f7d278e9951ba06002e1f778b7c272ed657a45474971f5" exitCode=0
Oct 29 22:39:27.230429 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:27.230411    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"
Oct 29 22:39:27.268722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:27.268692    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn" event={"ID":"2dd88602-acf5-41be-89f4-e5aa49e2a357","Type":"ContainerDied","Data":"3c229ca553c1469080f7d278e9951ba06002e1f778b7c272ed657a45474971f5"}
Oct 29 22:39:27.269875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:27.268923    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn" event={"ID":"2dd88602-acf5-41be-89f4-e5aa49e2a357","Type":"ContainerDied","Data":"5ec11230d189450edaf1b37f85788502fb53ae241c918af245b078ee5e827d69"}
Oct 29 22:39:27.269875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:27.268964    4559 scope.go:122] "RemoveContainer" containerID="3c229ca553c1469080f7d278e9951ba06002e1f778b7c272ed657a45474971f5"
Oct 29 22:39:27.282885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:27.282850    4559 scope.go:122] "RemoveContainer" containerID="3c229ca553c1469080f7d278e9951ba06002e1f778b7c272ed657a45474971f5"
Oct 29 22:39:27.283213 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:39:27.283188    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c229ca553c1469080f7d278e9951ba06002e1f778b7c272ed657a45474971f5\": container with ID starting with 3c229ca553c1469080f7d278e9951ba06002e1f778b7c272ed657a45474971f5 not found: ID does not exist" containerID="3c229ca553c1469080f7d278e9951ba06002e1f778b7c272ed657a45474971f5"
Oct 29 22:39:27.283315 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:27.283219    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c229ca553c1469080f7d278e9951ba06002e1f778b7c272ed657a45474971f5"} err="failed to get container status \"3c229ca553c1469080f7d278e9951ba06002e1f778b7c272ed657a45474971f5\": rpc error: code = NotFound desc = could not find container \"3c229ca553c1469080f7d278e9951ba06002e1f778b7c272ed657a45474971f5\": container with ID starting with 3c229ca553c1469080f7d278e9951ba06002e1f778b7c272ed657a45474971f5 not found: ID does not exist"
Oct 29 22:39:27.314841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:27.314789    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"]
Oct 29 22:39:27.327258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:27.327217    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee886f-588fb4f8b-rqldn"]
Oct 29 22:39:29.236330 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:29.236290    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd88602-acf5-41be-89f4-e5aa49e2a357" path="/var/lib/kubelet/pods/2dd88602-acf5-41be-89f4-e5aa49e2a357/volumes"
Oct 29 22:39:33.367643 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.367604    4559 generic.go:356] "Generic (PLEG): container finished" podID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" containerID="f230b8fda8326c414997183af033d54402c09d6b13c91bb439562b079482c444" exitCode=0
Oct 29 22:39:33.368164 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.367683    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm" event={"ID":"158bafa6-a6ed-49d4-8ac2-2722c7be5efb","Type":"ContainerDied","Data":"f230b8fda8326c414997183af033d54402c09d6b13c91bb439562b079482c444"}
Oct 29 22:39:33.368268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.368250    4559 scope.go:122] "RemoveContainer" containerID="f230b8fda8326c414997183af033d54402c09d6b13c91bb439562b079482c444"
Oct 29 22:39:33.370273 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.370241    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:39:33.370385 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.370329    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:39:33.370385 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.370380    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:39:33.695892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.695858    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"]
Oct 29 22:39:33.702880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.700666    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2dd88602-acf5-41be-89f4-e5aa49e2a357" containerName="payload-processing"
Oct 29 22:39:33.702880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.700699    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd88602-acf5-41be-89f4-e5aa49e2a357" containerName="payload-processing"
Oct 29 22:39:33.702880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.700736    4559 cpu_manager.go:414] "containerMap: removing container" podUID="89e9567e-9768-4800-8fee-cbe5b93d7850" containerName="payload-pre-processing"
Oct 29 22:39:33.702880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.700745    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e9567e-9768-4800-8fee-cbe5b93d7850" containerName="payload-pre-processing"
Oct 29 22:39:33.702880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.700758    4559 cpu_manager.go:414] "containerMap: removing container" podUID="63a99d09-6876-4624-99f0-3fb56f3b3935" containerName="payload-processing"
Oct 29 22:39:33.702880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.700766    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a99d09-6876-4624-99f0-3fb56f3b3935" containerName="payload-processing"
Oct 29 22:39:33.702880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.701096    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd88602-acf5-41be-89f4-e5aa49e2a357" containerName="payload-processing"
Oct 29 22:39:33.702880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.701117    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e9567e-9768-4800-8fee-cbe5b93d7850" containerName="payload-pre-processing"
Oct 29 22:39:33.702880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.701129    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a99d09-6876-4624-99f0-3fb56f3b3935" containerName="payload-processing"
Oct 29 22:39:33.709735 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.706399    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"]
Oct 29 22:39:33.709735 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.706516    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.722846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.722803    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c296a3-openshift-default-dockercfg-p5jk9\""
Oct 29 22:39:33.826449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.826408    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-istio-envoy\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.826449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.826454    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65fe44fa-7821-4581-9607-05d919e94273-istio-token\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.826753 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.826728    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65fe44fa-7821-4581-9607-05d919e94273-istio-podinfo\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.826922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.826903    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrtfw\" (UniqueName: \"kubernetes.io/projected/65fe44fa-7821-4581-9607-05d919e94273-kube-api-access-wrtfw\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.826971 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.826938    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-workload-socket\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.827007 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.826989    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-istio-data\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.827050 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.827040    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65fe44fa-7821-4581-9607-05d919e94273-istiod-ca-cert\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.827089 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.827070    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-workload-certs\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.827142 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.827114    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-credential-socket\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.941834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.933933    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65fe44fa-7821-4581-9607-05d919e94273-istio-podinfo\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.941834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.934054    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtfw\" (UniqueName: \"kubernetes.io/projected/65fe44fa-7821-4581-9607-05d919e94273-kube-api-access-wrtfw\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.941834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.934089    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-workload-socket\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.941834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.934127    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-istio-data\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.941834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.934183    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65fe44fa-7821-4581-9607-05d919e94273-istiod-ca-cert\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.941834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.934219    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-workload-certs\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.941834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.934254    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-credential-socket\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.941834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.934378    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-istio-envoy\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.941834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.934411    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65fe44fa-7821-4581-9607-05d919e94273-istio-token\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.941834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.936081    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65fe44fa-7821-4581-9607-05d919e94273-istiod-ca-cert\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.941834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.936610    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-workload-socket\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.950937 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.945597    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-credential-socket\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.950937 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.945873    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-workload-certs\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.957843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.956842    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-istio-data\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.966838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.963944    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65fe44fa-7821-4581-9607-05d919e94273-istio-token\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.992835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.992177    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-istio-envoy\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.993036 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.992893    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65fe44fa-7821-4581-9607-05d919e94273-istio-podinfo\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:33.997833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:33.993559    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrtfw\" (UniqueName: \"kubernetes.io/projected/65fe44fa-7821-4581-9607-05d919e94273-kube-api-access-wrtfw\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:34.033568 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:34.032253    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:34.383743 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:34.383704    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm" event={"ID":"158bafa6-a6ed-49d4-8ac2-2722c7be5efb","Type":"ContainerStarted","Data":"9dd62e009a5921088740e5c1727fd5a7cb015c0b386742ecaf8e17ceb7c4646f"}
Oct 29 22:39:34.645891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:34.645721    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"]
Oct 29 22:39:34.650206 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:39:34.650172    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65fe44fa_7821_4581_9607_05d919e94273.slice/crio-d5d0c4082608562160042b2491e7f408410bdf1a4e73c47eccea8efff323a076 WatchSource:0}: Error finding container d5d0c4082608562160042b2491e7f408410bdf1a4e73c47eccea8efff323a076: Status 404 returned error can't find the container with id d5d0c4082608562160042b2491e7f408410bdf1a4e73c47eccea8efff323a076
Oct 29 22:39:34.653384 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:34.653348    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:39:34.653477 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:34.653430    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:39:34.653477 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:34.653468    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:39:34.681624 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:34.681598    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:39:34.683410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:34.683386    4559 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 29 22:39:34.683542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:34.683442    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm" podUID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 29 22:39:34.917800 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:34.917712    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:39:35.393343 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:35.393309    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4" event={"ID":"65fe44fa-7821-4581-9607-05d919e94273","Type":"ContainerStarted","Data":"8524e8f9021ce677259bd2a7be6982af38b91d4322f62867409f352f8e3de2a8"}
Oct 29 22:39:35.393758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:35.393392    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4" event={"ID":"65fe44fa-7821-4581-9607-05d919e94273","Type":"ContainerStarted","Data":"d5d0c4082608562160042b2491e7f408410bdf1a4e73c47eccea8efff323a076"}
Oct 29 22:39:35.467858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:35.467673    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4" podStartSLOduration=2.467655139 podStartE2EDuration="2.467655139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:35.456146795 +0000 UTC m=+1514.367590360" watchObservedRunningTime="2026-10-29 22:39:35.467655139 +0000 UTC m=+1514.379098694"
Oct 29 22:39:35.686000 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:35.685892    4559 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 29 22:39:35.686503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:35.686465    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm" podUID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 29 22:39:36.032762 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:36.032679    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:36.065843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:36.065296    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:36.400964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:36.400933    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:36.401788 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:36.401767    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:39:36.682321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:36.682231    4559 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 29 22:39:36.682321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:36.682292    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm" podUID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 29 22:39:37.680915 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:37.680877    4559 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 29 22:39:37.681332 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:37.680943    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm" podUID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 29 22:39:38.680735 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:38.680691    4559 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 29 22:39:38.680960 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:38.680748    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm" podUID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 29 22:39:39.680961 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:39.680926    4559 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 29 22:39:39.681649 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:39.681617    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm" podUID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 29 22:39:39.804677 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:39.804641    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"]
Oct 29 22:39:39.811957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:39.811924    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"
Oct 29 22:39:39.826321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:39.826293    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"]
Oct 29 22:39:39.900933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:39.900897    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c296a3\""
Oct 29 22:39:39.901207 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:39.901135    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c296a3-dockercfg-7ckpn\""
Oct 29 22:39:39.956891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:39.950480    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj5pw\" (UniqueName: \"kubernetes.io/projected/83c4474e-4311-4cc6-a79f-aa4bc4edfcb0-kube-api-access-pj5pw\") pod \"payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg\" (UID: \"83c4474e-4311-4cc6-a79f-aa4bc4edfcb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"
Oct 29 22:39:39.956891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:39.950579    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c4474e-4311-4cc6-a79f-aa4bc4edfcb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg\" (UID: \"83c4474e-4311-4cc6-a79f-aa4bc4edfcb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"
Oct 29 22:39:40.246136 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:40.246034    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj5pw\" (UniqueName: \"kubernetes.io/projected/83c4474e-4311-4cc6-a79f-aa4bc4edfcb0-kube-api-access-pj5pw\") pod \"payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg\" (UID: \"83c4474e-4311-4cc6-a79f-aa4bc4edfcb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"
Oct 29 22:39:40.313008 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:40.312970    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c4474e-4311-4cc6-a79f-aa4bc4edfcb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg\" (UID: \"83c4474e-4311-4cc6-a79f-aa4bc4edfcb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"
Oct 29 22:39:40.333089 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:40.314478    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c4474e-4311-4cc6-a79f-aa4bc4edfcb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg\" (UID: \"83c4474e-4311-4cc6-a79f-aa4bc4edfcb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"
Oct 29 22:39:40.365924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:40.358780    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj5pw\" (UniqueName: \"kubernetes.io/projected/83c4474e-4311-4cc6-a79f-aa4bc4edfcb0-kube-api-access-pj5pw\") pod \"payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg\" (UID: \"83c4474e-4311-4cc6-a79f-aa4bc4edfcb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"
Oct 29 22:39:40.393216 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:40.393173    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"
Oct 29 22:39:40.682287 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:40.682249    4559 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 29 22:39:40.682914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:40.682884    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm" podUID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 29 22:39:40.888295 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:40.863588    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"]
Oct 29 22:39:41.496351 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:41.496251    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg" event={"ID":"83c4474e-4311-4cc6-a79f-aa4bc4edfcb0","Type":"ContainerStarted","Data":"955dac93fb7a24f9d879fea41427f2aef7e1fbd8ee1b17ad7a8fa70f2af9eb89"}
Oct 29 22:39:41.496537 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:41.496355    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg" event={"ID":"83c4474e-4311-4cc6-a79f-aa4bc4edfcb0","Type":"ContainerStarted","Data":"2cc3d523f8b8322f8c05f7035eba05083a9536d205505a7bb8d7a76a2e6c8ce2"}
Oct 29 22:39:41.496617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:41.496567    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"
Oct 29 22:39:41.538729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:41.538676    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg" podStartSLOduration=2.5386614180000002 podStartE2EDuration="2.538661418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:41.535668928 +0000 UTC m=+1520.447112484" watchObservedRunningTime="2026-10-29 22:39:41.538661418 +0000 UTC m=+1520.450104972"
Oct 29 22:39:41.732842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:41.727399    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:39:41.756268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:41.755204    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:39:43.697507 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:39:43.697467    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:39:49.168137 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:49.168098    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9098-820d83"]
Oct 29 22:39:49.228835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:49.221745    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9098-820d83"]
Oct 29 22:39:49.228835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:49.222944    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9098-820d83"
Oct 29 22:39:49.255872 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:49.242299    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xl57t\""
Oct 29 22:39:49.303334 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:49.303292    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5tbd\" (UniqueName: \"kubernetes.io/projected/c28453e4-82a8-4bab-8fe6-d80781cfffd3-kube-api-access-p5tbd\") pod \"test-curl-9098-820d83\" (UID: \"c28453e4-82a8-4bab-8fe6-d80781cfffd3\") " pod="opendatahub/test-curl-9098-820d83"
Oct 29 22:39:49.405583 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:49.405535    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5tbd\" (UniqueName: \"kubernetes.io/projected/c28453e4-82a8-4bab-8fe6-d80781cfffd3-kube-api-access-p5tbd\") pod \"test-curl-9098-820d83\" (UID: \"c28453e4-82a8-4bab-8fe6-d80781cfffd3\") " pod="opendatahub/test-curl-9098-820d83"
Oct 29 22:39:49.463886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:49.463365    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5tbd\" (UniqueName: \"kubernetes.io/projected/c28453e4-82a8-4bab-8fe6-d80781cfffd3-kube-api-access-p5tbd\") pod \"test-curl-9098-820d83\" (UID: \"c28453e4-82a8-4bab-8fe6-d80781cfffd3\") " pod="opendatahub/test-curl-9098-820d83"
Oct 29 22:39:49.588962 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:49.588933    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9098-820d83"
Oct 29 22:39:49.928488 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:39:49.928452    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc28453e4_82a8_4bab_8fe6_d80781cfffd3.slice/crio-7132c30b966a793120438109ba528bb149d2adbf91466cd8c4b47bdab7cd82bd WatchSource:0}: Error finding container 7132c30b966a793120438109ba528bb149d2adbf91466cd8c4b47bdab7cd82bd: Status 404 returned error can't find the container with id 7132c30b966a793120438109ba528bb149d2adbf91466cd8c4b47bdab7cd82bd
Oct 29 22:39:49.928623 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:49.928584    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9098-820d83"]
Oct 29 22:39:50.792607 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:50.792567    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9098-820d83" event={"ID":"c28453e4-82a8-4bab-8fe6-d80781cfffd3","Type":"ContainerStarted","Data":"d760ba71aca64965fc6ade1e142eff6fb3273857cf6b2bd0bca0b19a7e930740"}
Oct 29 22:39:50.793077 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:50.792617    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9098-820d83" event={"ID":"c28453e4-82a8-4bab-8fe6-d80781cfffd3","Type":"ContainerStarted","Data":"7132c30b966a793120438109ba528bb149d2adbf91466cd8c4b47bdab7cd82bd"}
Oct 29 22:39:51.896935 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:51.896880    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9098-820d83" podStartSLOduration=2.896864496 podStartE2EDuration="2.896864496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:50.888655787 +0000 UTC m=+1529.800099378" watchObservedRunningTime="2026-10-29 22:39:51.896864496 +0000 UTC m=+1530.808308049"
Oct 29 22:39:51.899159 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:51.899116    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9098-820d83"]
Oct 29 22:39:51.904621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:51.904581    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9098-820d83"]
Oct 29 22:39:52.508181 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:52.508148    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"
Oct 29 22:39:53.176840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:53.174901    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9098-820d83" podUID="c28453e4-82a8-4bab-8fe6-d80781cfffd3" containerName="test-curl-9098-820d83" containerID="cri-o://d760ba71aca64965fc6ade1e142eff6fb3273857cf6b2bd0bca0b19a7e930740" gracePeriod=2
Oct 29 22:39:53.201105 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:39:53.201059    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9098-820d83\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="c28453e4-82a8-4bab-8fe6-d80781cfffd3" pod="opendatahub/test-curl-9098-820d83"
Oct 29 22:39:55.750658 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:55.750626    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9098-820d83"
Oct 29 22:39:55.838973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:55.838929    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c28453e4-82a8-4bab-8fe6-d80781cfffd3-kube-api-access-p5tbd\" (UniqueName: \"kubernetes.io/projected/c28453e4-82a8-4bab-8fe6-d80781cfffd3-kube-api-access-p5tbd\") pod \"c28453e4-82a8-4bab-8fe6-d80781cfffd3\" (UID: \"c28453e4-82a8-4bab-8fe6-d80781cfffd3\") "
Oct 29 22:39:55.841362 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:55.841330    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c28453e4-82a8-4bab-8fe6-d80781cfffd3-kube-api-access-p5tbd" pod "c28453e4-82a8-4bab-8fe6-d80781cfffd3" (UID: "c28453e4-82a8-4bab-8fe6-d80781cfffd3"). InnerVolumeSpecName "kube-api-access-p5tbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:55.941158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:55.941115    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5tbd\" (UniqueName: \"kubernetes.io/projected/c28453e4-82a8-4bab-8fe6-d80781cfffd3-kube-api-access-p5tbd\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:39:56.179632 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:56.179592    4559 generic.go:356] "Generic (PLEG): container finished" podID="c28453e4-82a8-4bab-8fe6-d80781cfffd3" containerID="d760ba71aca64965fc6ade1e142eff6fb3273857cf6b2bd0bca0b19a7e930740" exitCode=137
Oct 29 22:39:56.179868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:56.179636    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9098-820d83"
Oct 29 22:39:56.179868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:56.179705    4559 scope.go:122] "RemoveContainer" containerID="d760ba71aca64965fc6ade1e142eff6fb3273857cf6b2bd0bca0b19a7e930740"
Oct 29 22:39:56.191039 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:56.191010    4559 scope.go:122] "RemoveContainer" containerID="d760ba71aca64965fc6ade1e142eff6fb3273857cf6b2bd0bca0b19a7e930740"
Oct 29 22:39:56.191433 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:39:56.191408    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d760ba71aca64965fc6ade1e142eff6fb3273857cf6b2bd0bca0b19a7e930740\": container with ID starting with d760ba71aca64965fc6ade1e142eff6fb3273857cf6b2bd0bca0b19a7e930740 not found: ID does not exist" containerID="d760ba71aca64965fc6ade1e142eff6fb3273857cf6b2bd0bca0b19a7e930740"
Oct 29 22:39:56.191531 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:56.191444    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d760ba71aca64965fc6ade1e142eff6fb3273857cf6b2bd0bca0b19a7e930740"} err="failed to get container status \"d760ba71aca64965fc6ade1e142eff6fb3273857cf6b2bd0bca0b19a7e930740\": rpc error: code = NotFound desc = could not find container \"d760ba71aca64965fc6ade1e142eff6fb3273857cf6b2bd0bca0b19a7e930740\": container with ID starting with d760ba71aca64965fc6ade1e142eff6fb3273857cf6b2bd0bca0b19a7e930740 not found: ID does not exist"
Oct 29 22:39:56.204068 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:39:56.204024    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9098-820d83\" is forbidden: User \"system:node:ip-10-0-138-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-2.ec2.internal' and this object" podUID="c28453e4-82a8-4bab-8fe6-d80781cfffd3" pod="opendatahub/test-curl-9098-820d83"
Oct 29 22:39:57.234525 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.234485    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c28453e4-82a8-4bab-8fe6-d80781cfffd3" path="/var/lib/kubelet/pods/c28453e4-82a8-4bab-8fe6-d80781cfffd3/volumes"
Oct 29 22:39:57.489209 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.489063    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"]
Oct 29 22:39:57.490094 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.490071    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c28453e4-82a8-4bab-8fe6-d80781cfffd3" containerName="test-curl-9098-820d83"
Oct 29 22:39:57.490094 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.490092    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c28453e4-82a8-4bab-8fe6-d80781cfffd3" containerName="test-curl-9098-820d83"
Oct 29 22:39:57.490246 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.490237    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c28453e4-82a8-4bab-8fe6-d80781cfffd3" containerName="test-curl-9098-820d83"
Oct 29 22:39:57.495995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.495971    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.512444 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.512384    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1fa4b8b6-openshift-default-dockercfg-mnpx6\""
Oct 29 22:39:57.532711 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.532674    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"]
Oct 29 22:39:57.566675 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.566621    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.567044 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.566696    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.567044 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.566854    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.567044 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.566975    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.567268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.567142    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h9dw\" (UniqueName: \"kubernetes.io/projected/4042e16a-2f51-4588-b54c-bfce92dd57cc-kube-api-access-2h9dw\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.567268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.567187    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.567373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.567286    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4042e16a-2f51-4588-b54c-bfce92dd57cc-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.567434 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.567370    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.567434 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.567412    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.668975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.668937    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h9dw\" (UniqueName: \"kubernetes.io/projected/4042e16a-2f51-4588-b54c-bfce92dd57cc-kube-api-access-2h9dw\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.669193 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.668993    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.669193 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.669085    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4042e16a-2f51-4588-b54c-bfce92dd57cc-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.669193 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.669125    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.669193 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.669156    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.669466 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.669268    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.669466 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.669315    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.669466 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.669345    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.669466 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.669443    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.670416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.669678    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.670416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.669790    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.670416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.670277    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.670416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.670373    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4042e16a-2f51-4588-b54c-bfce92dd57cc-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.670764 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.670609    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.672389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.672362    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.673308 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.673283    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.689699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.689666    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.689929 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.689786    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h9dw\" (UniqueName: \"kubernetes.io/projected/4042e16a-2f51-4588-b54c-bfce92dd57cc-kube-api-access-2h9dw\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:57.814994 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:57.814907    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:58.153606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:58.153573    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"]
Oct 29 22:39:58.155118 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:39:58.155090    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4042e16a_2f51_4588_b54c_bfce92dd57cc.slice/crio-547a56e34db6d78bf2376446b3829774cb752a4b5667580fe65f1357825bdd73 WatchSource:0}: Error finding container 547a56e34db6d78bf2376446b3829774cb752a4b5667580fe65f1357825bdd73: Status 404 returned error can't find the container with id 547a56e34db6d78bf2376446b3829774cb752a4b5667580fe65f1357825bdd73
Oct 29 22:39:58.159591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:58.159550    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:39:58.159687 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:58.159648    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:39:58.159727 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:58.159693    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:39:58.209606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:58.209571    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh" event={"ID":"4042e16a-2f51-4588-b54c-bfce92dd57cc","Type":"ContainerStarted","Data":"547a56e34db6d78bf2376446b3829774cb752a4b5667580fe65f1357825bdd73"}
Oct 29 22:39:59.219286 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:59.219249    4559 generic.go:356] "Generic (PLEG): container finished" podID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" containerID="92c0f28d138be4dfb0bfb677c31b3358b72fc95c64844dfb794afc6fa804130a" exitCode=0
Oct 29 22:39:59.219921 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:59.219325    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt" event={"ID":"71bc892b-1fac-4c55-a4c4-493bf662a9ae","Type":"ContainerDied","Data":"92c0f28d138be4dfb0bfb677c31b3358b72fc95c64844dfb794afc6fa804130a"}
Oct 29 22:39:59.220282 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:59.220262    4559 scope.go:122] "RemoveContainer" containerID="92c0f28d138be4dfb0bfb677c31b3358b72fc95c64844dfb794afc6fa804130a"
Oct 29 22:39:59.222260 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:59.222228    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:39:59.222337 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:59.222309    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:39:59.222374 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:59.222348    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:39:59.231021 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:59.230983    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh" event={"ID":"4042e16a-2f51-4588-b54c-bfce92dd57cc","Type":"ContainerStarted","Data":"55a48a5177e24c35a5eb9f74e02da0460cab9012197a0fb1ea4c63042803d8bf"}
Oct 29 22:39:59.329987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:59.329918    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh" podStartSLOduration=2.329886449 podStartE2EDuration="2.329886449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:59.327211491 +0000 UTC m=+1538.238655066" watchObservedRunningTime="2026-10-29 22:39:59.329886449 +0000 UTC m=+1538.241330003"
Oct 29 22:39:59.815449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:59.815413    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:39:59.817248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:59.817218    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 29 22:39:59.817378 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:39:59.817280    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh" podUID="4042e16a-2f51-4588-b54c-bfce92dd57cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 29 22:40:00.311155 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:00.311115    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt" event={"ID":"71bc892b-1fac-4c55-a4c4-493bf662a9ae","Type":"ContainerStarted","Data":"2e680dc60deea7f8d4bcd762190f29259215e753b51099c2451b96c37d1aadf0"}
Oct 29 22:40:00.324137 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:00.324105    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:40:00.325150 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:00.325123    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 29 22:40:00.325300 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:00.325179    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt" podUID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 29 22:40:00.924418 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:00.924378    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 29 22:40:00.924697 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:00.924670    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh" podUID="4042e16a-2f51-4588-b54c-bfce92dd57cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 29 22:40:01.384103 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:01.383957    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 29 22:40:01.384103 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:01.384030    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt" podUID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 29 22:40:01.553873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:01.535314    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"]
Oct 29 22:40:01.575837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:01.555058    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerName="istio-proxy" containerID="cri-o://45bd044f8a32aeabe74230c5220b8782c00d68c898db7f9f11fd202019031dff" gracePeriod=30
Oct 29 22:40:01.605192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:01.596393    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:40:01.760946 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:01.755071    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:40:01.881915 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:01.878155    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 29 22:40:01.881915 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:01.878221    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh" podUID="4042e16a-2f51-4588-b54c-bfce92dd57cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 29 22:40:02.163837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:02.157971    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:40:02.250840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:02.249942    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:40:02.328830 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:02.295344    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:40:02.332984 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:02.332949    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 29 22:40:02.410060 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:02.409929    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt" podUID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 29 22:40:02.424966 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:02.363534    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:40:02.615036 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:02.614998    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:40:02.817503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:02.817397    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 29 22:40:02.817758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:02.817729    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh" podUID="4042e16a-2f51-4588-b54c-bfce92dd57cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 29 22:40:03.284277 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:03.284241    4559 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:40:03.284556 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:03.284529    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:40:03.343150 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:03.343109    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 29 22:40:03.343668 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:03.343630    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt" podUID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 29 22:40:03.826835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:03.826708    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 29 22:40:03.826835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:03.826794    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh" podUID="4042e16a-2f51-4588-b54c-bfce92dd57cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 29 22:40:04.343998 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:04.343958    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 29 22:40:04.344389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:04.344346    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt" podUID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 29 22:40:04.816153 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:04.816113    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 29 22:40:04.816337 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:04.816175    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh" podUID="4042e16a-2f51-4588-b54c-bfce92dd57cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 29 22:40:05.324960 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:05.324927    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 29 22:40:05.325373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:05.324984    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt" podUID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 29 22:40:05.816241 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:05.816207    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 29 22:40:05.816563 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:05.816518    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh" podUID="4042e16a-2f51-4588-b54c-bfce92dd57cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 29 22:40:06.324832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:06.324789    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 29 22:40:06.325076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:06.325054    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt" podUID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 29 22:40:06.796032 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:06.795994    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"]
Oct 29 22:40:06.796523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:06.796488    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2" podUID="cd055ebb-b30b-447d-831d-1ff95c57dc55" containerName="istio-proxy" containerID="cri-o://0f43e371f53cc199d3fd92e97bccd6c9e2c18f01f100998e4b9e6f9c59a92cd9" gracePeriod=30
Oct 29 22:40:06.816338 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:06.816300    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 29 22:40:06.816517 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:06.816363    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh" podUID="4042e16a-2f51-4588-b54c-bfce92dd57cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 29 22:40:07.008887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.008739    4559 generic.go:356] "Generic (PLEG): container finished" podID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerID="45bd044f8a32aeabe74230c5220b8782c00d68c898db7f9f11fd202019031dff" exitCode=0
Oct 29 22:40:07.009586 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.009555    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" event={"ID":"bf5762b8-dbb2-4967-a9c5-8a2876e913fb","Type":"ContainerDied","Data":"45bd044f8a32aeabe74230c5220b8782c00d68c898db7f9f11fd202019031dff"}
Oct 29 22:40:07.009883 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.009865    4559 scope.go:122] "RemoveContainer" containerID="5fbc65ca6eaab59555eb417db20e1b9251264228558ca765d8a14f9f167efae2"
Oct 29 22:40:07.123053 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.123020    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:40:07.318373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.318268    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-kube-api-access-95km4\" (UniqueName: \"kubernetes.io/projected/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-kube-api-access-95km4\") pod \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") "
Oct 29 22:40:07.318713 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.318694    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-credential-socket\") pod \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") "
Oct 29 22:40:07.318913 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.318898    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-envoy\") pod \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") "
Oct 29 22:40:07.319128 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.319115    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-workload-socket\") pod \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") "
Oct 29 22:40:07.319240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.319227    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-token\" (UniqueName: \"kubernetes.io/projected/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-token\") pod \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") "
Oct 29 22:40:07.319409 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.319395    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-data\") pod \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") "
Oct 29 22:40:07.319551 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.319537    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-workload-certs\") pod \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") "
Oct 29 22:40:07.319682 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.319668    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-podinfo\") pod \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") "
Oct 29 22:40:07.319967 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.319952    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istiod-ca-cert\") pod \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\" (UID: \"bf5762b8-dbb2-4967-a9c5-8a2876e913fb\") "
Oct 29 22:40:07.324174 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.324143    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-credential-socket" pod "bf5762b8-dbb2-4967-a9c5-8a2876e913fb" (UID: "bf5762b8-dbb2-4967-a9c5-8a2876e913fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:07.331112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.324883    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-data" pod "bf5762b8-dbb2-4967-a9c5-8a2876e913fb" (UID: "bf5762b8-dbb2-4967-a9c5-8a2876e913fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:07.331112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.325106    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-workload-certs" pod "bf5762b8-dbb2-4967-a9c5-8a2876e913fb" (UID: "bf5762b8-dbb2-4967-a9c5-8a2876e913fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:07.331112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.327145    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-workload-socket" pod "bf5762b8-dbb2-4967-a9c5-8a2876e913fb" (UID: "bf5762b8-dbb2-4967-a9c5-8a2876e913fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:07.331112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.328508    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istiod-ca-cert" pod "bf5762b8-dbb2-4967-a9c5-8a2876e913fb" (UID: "bf5762b8-dbb2-4967-a9c5-8a2876e913fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:07.331112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.329083    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 29 22:40:07.331112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.329142    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt" podUID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 29 22:40:07.332720 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.332458    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-token" pod "bf5762b8-dbb2-4967-a9c5-8a2876e913fb" (UID: "bf5762b8-dbb2-4967-a9c5-8a2876e913fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:07.389676 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.389634    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-kube-api-access-95km4" pod "bf5762b8-dbb2-4967-a9c5-8a2876e913fb" (UID: "bf5762b8-dbb2-4967-a9c5-8a2876e913fb"). InnerVolumeSpecName "kube-api-access-95km4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:07.406471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.405129    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-podinfo" pod "bf5762b8-dbb2-4967-a9c5-8a2876e913fb" (UID: "bf5762b8-dbb2-4967-a9c5-8a2876e913fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:40:07.413031 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.412985    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-envoy" pod "bf5762b8-dbb2-4967-a9c5-8a2876e913fb" (UID: "bf5762b8-dbb2-4967-a9c5-8a2876e913fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:07.426139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.425256    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95km4\" (UniqueName: \"kubernetes.io/projected/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-kube-api-access-95km4\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:07.426139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.425292    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:07.426139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.425309    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:07.426139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.425325    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:07.426139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.425337    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:07.426139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.425349    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:07.426139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.425361    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:07.426139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.425373    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:07.426139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:07.425386    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5762b8-dbb2-4967-a9c5-8a2876e913fb-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:08.019092 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:08.019009    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"
Oct 29 22:40:08.019092 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:08.019030    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6" event={"ID":"bf5762b8-dbb2-4967-a9c5-8a2876e913fb","Type":"ContainerDied","Data":"3eb0c2d3b71c4659c96383d99b5a2d30c366a3c8009be006f4606adf0f2b9f86"}
Oct 29 22:40:08.019321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:08.019101    4559 scope.go:122] "RemoveContainer" containerID="45bd044f8a32aeabe74230c5220b8782c00d68c898db7f9f11fd202019031dff"
Oct 29 22:40:08.094787 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:08.091357    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"]
Oct 29 22:40:08.107785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:08.107728    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-x82q6"]
Oct 29 22:40:08.282423 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:08.282338    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:40:08.282794 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:08.282776    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:40:08.283899 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:08.283878    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:40:08.330436 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:08.330405    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:40:08.331474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:08.331451    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:40:08.842469 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:08.842434    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"]
Oct 29 22:40:08.842714 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:08.842689    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerName="istio-proxy" containerID="cri-o://16e3a9277206cd2b7f0b17cc9fbab78fcb17bee92bb2788863dea3bceb5fc985" gracePeriod=30
Oct 29 22:40:09.322351 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:09.315260    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" path="/var/lib/kubelet/pods/bf5762b8-dbb2-4967-a9c5-8a2876e913fb/volumes"
Oct 29 22:40:12.136674 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.136644    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:40:12.137370 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.137335    4559 generic.go:356] "Generic (PLEG): container finished" podID="cd055ebb-b30b-447d-831d-1ff95c57dc55" containerID="0f43e371f53cc199d3fd92e97bccd6c9e2c18f01f100998e4b9e6f9c59a92cd9" exitCode=0
Oct 29 22:40:12.137499 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.137442    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2" event={"ID":"cd055ebb-b30b-447d-831d-1ff95c57dc55","Type":"ContainerDied","Data":"0f43e371f53cc199d3fd92e97bccd6c9e2c18f01f100998e4b9e6f9c59a92cd9"}
Oct 29 22:40:12.137568 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.137522    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2" event={"ID":"cd055ebb-b30b-447d-831d-1ff95c57dc55","Type":"ContainerDied","Data":"d3f02b1407e25e12cea401d385c42a2fe7bedf5cd87b73a6a0be05023407112e"}
Oct 29 22:40:12.137568 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.137553    4559 scope.go:122] "RemoveContainer" containerID="0f43e371f53cc199d3fd92e97bccd6c9e2c18f01f100998e4b9e6f9c59a92cd9"
Oct 29 22:40:12.152008 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.151984    4559 scope.go:122] "RemoveContainer" containerID="0f43e371f53cc199d3fd92e97bccd6c9e2c18f01f100998e4b9e6f9c59a92cd9"
Oct 29 22:40:12.152454 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:40:12.152406    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f43e371f53cc199d3fd92e97bccd6c9e2c18f01f100998e4b9e6f9c59a92cd9\": container with ID starting with 0f43e371f53cc199d3fd92e97bccd6c9e2c18f01f100998e4b9e6f9c59a92cd9 not found: ID does not exist" containerID="0f43e371f53cc199d3fd92e97bccd6c9e2c18f01f100998e4b9e6f9c59a92cd9"
Oct 29 22:40:12.152560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.152448    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f43e371f53cc199d3fd92e97bccd6c9e2c18f01f100998e4b9e6f9c59a92cd9"} err="failed to get container status \"0f43e371f53cc199d3fd92e97bccd6c9e2c18f01f100998e4b9e6f9c59a92cd9\": rpc error: code = NotFound desc = could not find container \"0f43e371f53cc199d3fd92e97bccd6c9e2c18f01f100998e4b9e6f9c59a92cd9\": container with ID starting with 0f43e371f53cc199d3fd92e97bccd6c9e2c18f01f100998e4b9e6f9c59a92cd9 not found: ID does not exist"
Oct 29 22:40:12.213064 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.213015    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-envoy\") pod \"cd055ebb-b30b-447d-831d-1ff95c57dc55\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") "
Oct 29 22:40:12.213269 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.213125    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-podinfo\") pod \"cd055ebb-b30b-447d-831d-1ff95c57dc55\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") "
Oct 29 22:40:12.213269 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.213163    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-data\") pod \"cd055ebb-b30b-447d-831d-1ff95c57dc55\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") "
Oct 29 22:40:12.213269 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.213262    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-workload-certs\") pod \"cd055ebb-b30b-447d-831d-1ff95c57dc55\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") "
Oct 29 22:40:12.213446 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.213330    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-token\" (UniqueName: \"kubernetes.io/projected/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-token\") pod \"cd055ebb-b30b-447d-831d-1ff95c57dc55\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") "
Oct 29 22:40:12.213446 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.213365    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-workload-socket\") pod \"cd055ebb-b30b-447d-831d-1ff95c57dc55\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") "
Oct 29 22:40:12.213559 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.213452    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd055ebb-b30b-447d-831d-1ff95c57dc55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd055ebb-b30b-447d-831d-1ff95c57dc55-istiod-ca-cert\") pod \"cd055ebb-b30b-447d-831d-1ff95c57dc55\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") "
Oct 29 22:40:12.213559 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.213487    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-credential-socket\") pod \"cd055ebb-b30b-447d-831d-1ff95c57dc55\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") "
Oct 29 22:40:12.213559 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.213524    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd055ebb-b30b-447d-831d-1ff95c57dc55-kube-api-access-4z7fq\" (UniqueName: \"kubernetes.io/projected/cd055ebb-b30b-447d-831d-1ff95c57dc55-kube-api-access-4z7fq\") pod \"cd055ebb-b30b-447d-831d-1ff95c57dc55\" (UID: \"cd055ebb-b30b-447d-831d-1ff95c57dc55\") "
Oct 29 22:40:12.216790 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.216498    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-token" pod "cd055ebb-b30b-447d-831d-1ff95c57dc55" (UID: "cd055ebb-b30b-447d-831d-1ff95c57dc55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:12.217350 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.217025    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-envoy" pod "cd055ebb-b30b-447d-831d-1ff95c57dc55" (UID: "cd055ebb-b30b-447d-831d-1ff95c57dc55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:12.217515 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.217370    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd055ebb-b30b-447d-831d-1ff95c57dc55-istiod-ca-cert" pod "cd055ebb-b30b-447d-831d-1ff95c57dc55" (UID: "cd055ebb-b30b-447d-831d-1ff95c57dc55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:12.217658 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.217633    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-workload-socket" pod "cd055ebb-b30b-447d-831d-1ff95c57dc55" (UID: "cd055ebb-b30b-447d-831d-1ff95c57dc55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:12.217910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.217879    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-data" pod "cd055ebb-b30b-447d-831d-1ff95c57dc55" (UID: "cd055ebb-b30b-447d-831d-1ff95c57dc55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:12.218107 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.218083    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-credential-socket" pod "cd055ebb-b30b-447d-831d-1ff95c57dc55" (UID: "cd055ebb-b30b-447d-831d-1ff95c57dc55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:12.218315 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.218294    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-workload-certs" pod "cd055ebb-b30b-447d-831d-1ff95c57dc55" (UID: "cd055ebb-b30b-447d-831d-1ff95c57dc55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:12.221258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.221219    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-podinfo" pod "cd055ebb-b30b-447d-831d-1ff95c57dc55" (UID: "cd055ebb-b30b-447d-831d-1ff95c57dc55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:40:12.222300 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.222271    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd055ebb-b30b-447d-831d-1ff95c57dc55-kube-api-access-4z7fq" pod "cd055ebb-b30b-447d-831d-1ff95c57dc55" (UID: "cd055ebb-b30b-447d-831d-1ff95c57dc55"). InnerVolumeSpecName "kube-api-access-4z7fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:12.319474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.318327    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.319474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.318362    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.319474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.318376    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd055ebb-b30b-447d-831d-1ff95c57dc55-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.319474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.318390    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.319474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.318404    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z7fq\" (UniqueName: \"kubernetes.io/projected/cd055ebb-b30b-447d-831d-1ff95c57dc55-kube-api-access-4z7fq\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.319474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.318417    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.319474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.318431    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.319474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.318442    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.319474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.318455    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd055ebb-b30b-447d-831d-1ff95c57dc55-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.738723 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:12.737950    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"]
Oct 29 22:40:13.151082 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:13.151046    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"
Oct 29 22:40:13.213722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:13.213691    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"]
Oct 29 22:40:13.232868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:13.232836    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-pgwf2"]
Oct 29 22:40:13.894179 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:40:13.894057    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:40:14.179459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.179422    4559 generic.go:356] "Generic (PLEG): container finished" podID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerID="16e3a9277206cd2b7f0b17cc9fbab78fcb17bee92bb2788863dea3bceb5fc985" exitCode=0
Oct 29 22:40:14.179903 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.179712    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" event={"ID":"2040bbf0-5596-401b-b7f6-b3fa571c4d94","Type":"ContainerDied","Data":"16e3a9277206cd2b7f0b17cc9fbab78fcb17bee92bb2788863dea3bceb5fc985"}
Oct 29 22:40:14.179903 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.179781    4559 scope.go:122] "RemoveContainer" containerID="fe86c21f19f80948a127927d9d77da3248c1bd765ff7f602c4c0bd0b01f22b92"
Oct 29 22:40:14.190363 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.190334    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:40:14.254787 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.254759    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.320123    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-envoy\") pod \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") "
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.320186    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-workload-certs\") pod \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") "
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.320303    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-credential-socket\") pod \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") "
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.320353    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-workload-socket\") pod \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") "
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.320482    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2040bbf0-5596-401b-b7f6-b3fa571c4d94-kube-api-access-j24c4\" (UniqueName: \"kubernetes.io/projected/2040bbf0-5596-401b-b7f6-b3fa571c4d94-kube-api-access-j24c4\") pod \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") "
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.320584    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-data\") pod \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") "
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.320695    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istiod-ca-cert\") pod \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") "
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.320694    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-credential-socket" pod "2040bbf0-5596-401b-b7f6-b3fa571c4d94" (UID: "2040bbf0-5596-401b-b7f6-b3fa571c4d94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.320758    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-podinfo\") pod \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") "
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.320881    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-token\" (UniqueName: \"kubernetes.io/projected/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-token\") pod \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\" (UID: \"2040bbf0-5596-401b-b7f6-b3fa571c4d94\") "
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.320892    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-workload-socket" pod "2040bbf0-5596-401b-b7f6-b3fa571c4d94" (UID: "2040bbf0-5596-401b-b7f6-b3fa571c4d94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.321269    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istiod-ca-cert" pod "2040bbf0-5596-401b-b7f6-b3fa571c4d94" (UID: "2040bbf0-5596-401b-b7f6-b3fa571c4d94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.321525    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-data" pod "2040bbf0-5596-401b-b7f6-b3fa571c4d94" (UID: "2040bbf0-5596-401b-b7f6-b3fa571c4d94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.322158    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-workload-certs" pod "2040bbf0-5596-401b-b7f6-b3fa571c4d94" (UID: "2040bbf0-5596-401b-b7f6-b3fa571c4d94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.322176    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.322196    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.322210    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:14.322456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.322221    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:14.335997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.335865    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2040bbf0-5596-401b-b7f6-b3fa571c4d94-kube-api-access-j24c4" pod "2040bbf0-5596-401b-b7f6-b3fa571c4d94" (UID: "2040bbf0-5596-401b-b7f6-b3fa571c4d94"). InnerVolumeSpecName "kube-api-access-j24c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:14.335997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.335959    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-envoy" pod "2040bbf0-5596-401b-b7f6-b3fa571c4d94" (UID: "2040bbf0-5596-401b-b7f6-b3fa571c4d94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:14.335997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.335983    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-token" pod "2040bbf0-5596-401b-b7f6-b3fa571c4d94" (UID: "2040bbf0-5596-401b-b7f6-b3fa571c4d94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:14.337213 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.337190    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-podinfo" pod "2040bbf0-5596-401b-b7f6-b3fa571c4d94" (UID: "2040bbf0-5596-401b-b7f6-b3fa571c4d94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:40:14.449897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.443686    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:14.449897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.443729    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:14.449897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.443744    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:14.449897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.443756    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2040bbf0-5596-401b-b7f6-b3fa571c4d94-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:14.449897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:14.443771    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j24c4\" (UniqueName: \"kubernetes.io/projected/2040bbf0-5596-401b-b7f6-b3fa571c4d94-kube-api-access-j24c4\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:15.219740 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:15.219702    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"
Oct 29 22:40:15.233587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:15.233554    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd055ebb-b30b-447d-831d-1ff95c57dc55" path="/var/lib/kubelet/pods/cd055ebb-b30b-447d-831d-1ff95c57dc55/volumes"
Oct 29 22:40:15.234224 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:15.234200    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr" event={"ID":"2040bbf0-5596-401b-b7f6-b3fa571c4d94","Type":"ContainerDied","Data":"2ba7f3492cf1c8b567d55d228117523fbe0a6be6f8226c06cbfce4b44af214e4"}
Oct 29 22:40:15.234350 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:15.234233    4559 scope.go:122] "RemoveContainer" containerID="16e3a9277206cd2b7f0b17cc9fbab78fcb17bee92bb2788863dea3bceb5fc985"
Oct 29 22:40:15.294405 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:15.294371    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"]
Oct 29 22:40:15.298558 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:15.298531    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-616da7-openshift-default-85b49d64c5-6ctxr"]
Oct 29 22:40:16.376575 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:16.376544    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"]
Oct 29 22:40:16.377100 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:16.376760    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j" podUID="ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb" containerName="maas-api" containerID="cri-o://a4b0aa60b86b3b3444aa5a7120b6665765c7c18c75bc2e5abec677a24fc75be4" gracePeriod=30
Oct 29 22:40:16.550959 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:16.550919    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j" podUID="ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.133:8443/health\": dial tcp 10.134.0.133:8443: connect: connection refused"
Oct 29 22:40:17.123591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.123209    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:40:17.266570 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.266462    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-maas-api-tls\") pod \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\" (UID: \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\") "
Oct 29 22:40:17.266736 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.266571    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-kube-api-access-tlt88\" (UniqueName: \"kubernetes.io/projected/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-kube-api-access-tlt88\") pod \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\" (UID: \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\") "
Oct 29 22:40:17.266736 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.266650    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-openshift-service-ca\") pod \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\" (UID: \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\") "
Oct 29 22:40:17.266736 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.266728    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-metrics-certs\") pod \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\" (UID: \"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb\") "
Oct 29 22:40:17.285678 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.285635    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-openshift-service-ca" pod "ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb" (UID: "ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:17.299835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.289930    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-metrics-certs" pod "ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb" (UID: "ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:40:17.300909 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.300870    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" path="/var/lib/kubelet/pods/2040bbf0-5596-401b-b7f6-b3fa571c4d94/volumes"
Oct 29 22:40:17.309667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.303313    4559 generic.go:356] "Generic (PLEG): container finished" podID="ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb" containerID="a4b0aa60b86b3b3444aa5a7120b6665765c7c18c75bc2e5abec677a24fc75be4" exitCode=0
Oct 29 22:40:17.309667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.303430    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"
Oct 29 22:40:17.309667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.304541    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j" event={"ID":"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb","Type":"ContainerDied","Data":"a4b0aa60b86b3b3444aa5a7120b6665765c7c18c75bc2e5abec677a24fc75be4"}
Oct 29 22:40:17.309667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.304608    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j" event={"ID":"ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb","Type":"ContainerDied","Data":"8c50695992c7a7dd1e11ed60470496d2489689625d6e6cf040311198be687eed"}
Oct 29 22:40:17.309667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.304641    4559 scope.go:122] "RemoveContainer" containerID="a4b0aa60b86b3b3444aa5a7120b6665765c7c18c75bc2e5abec677a24fc75be4"
Oct 29 22:40:17.309667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.307844    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-kube-api-access-tlt88" pod "ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb" (UID: "ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb"). InnerVolumeSpecName "kube-api-access-tlt88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:17.309667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.308141    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-maas-api-tls" pod "ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb" (UID: "ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:40:17.339558 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.339531    4559 scope.go:122] "RemoveContainer" containerID="a4b0aa60b86b3b3444aa5a7120b6665765c7c18c75bc2e5abec677a24fc75be4"
Oct 29 22:40:17.340799 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:40:17.340728    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b0aa60b86b3b3444aa5a7120b6665765c7c18c75bc2e5abec677a24fc75be4\": container with ID starting with a4b0aa60b86b3b3444aa5a7120b6665765c7c18c75bc2e5abec677a24fc75be4 not found: ID does not exist" containerID="a4b0aa60b86b3b3444aa5a7120b6665765c7c18c75bc2e5abec677a24fc75be4"
Oct 29 22:40:17.340799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.340762    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b0aa60b86b3b3444aa5a7120b6665765c7c18c75bc2e5abec677a24fc75be4"} err="failed to get container status \"a4b0aa60b86b3b3444aa5a7120b6665765c7c18c75bc2e5abec677a24fc75be4\": rpc error: code = NotFound desc = could not find container \"a4b0aa60b86b3b3444aa5a7120b6665765c7c18c75bc2e5abec677a24fc75be4\": container with ID starting with a4b0aa60b86b3b3444aa5a7120b6665765c7c18c75bc2e5abec677a24fc75be4 not found: ID does not exist"
Oct 29 22:40:17.369431 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.369395    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:17.369431 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.369429    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:17.369696 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.369444    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlt88\" (UniqueName: \"kubernetes.io/projected/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-kube-api-access-tlt88\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:17.369696 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.369458    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:17.655642 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.655598    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"]
Oct 29 22:40:17.668022 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.667984    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa4b8b6-55c54f6df7-kmr6j"]
Oct 29 22:40:17.738372 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:17.738298    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st" podUID="666f8f93-509b-42a4-becd-e3b89ebe144b" containerName="payload-processing" containerID="cri-o://6d88454b805507be2fa6bcea0da8b6a38f6d734d74f92570462307e98dc19683" gracePeriod=25
Oct 29 22:40:18.060349 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:18.060322    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"
Oct 29 22:40:18.195712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:18.195683    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/666f8f93-509b-42a4-becd-e3b89ebe144b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666f8f93-509b-42a4-becd-e3b89ebe144b-plugins-config-volume\") pod \"666f8f93-509b-42a4-becd-e3b89ebe144b\" (UID: \"666f8f93-509b-42a4-becd-e3b89ebe144b\") "
Oct 29 22:40:18.196024 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:18.195998    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666f8f93-509b-42a4-becd-e3b89ebe144b-kube-api-access-z5gxx\" (UniqueName: \"kubernetes.io/projected/666f8f93-509b-42a4-becd-e3b89ebe144b-kube-api-access-z5gxx\") pod \"666f8f93-509b-42a4-becd-e3b89ebe144b\" (UID: \"666f8f93-509b-42a4-becd-e3b89ebe144b\") "
Oct 29 22:40:18.196190 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:18.196171    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/666f8f93-509b-42a4-becd-e3b89ebe144b-plugins-config-volume" pod "666f8f93-509b-42a4-becd-e3b89ebe144b" (UID: "666f8f93-509b-42a4-becd-e3b89ebe144b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:18.196720 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:18.196702    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666f8f93-509b-42a4-becd-e3b89ebe144b-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:18.198590 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:18.198564    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666f8f93-509b-42a4-becd-e3b89ebe144b-kube-api-access-z5gxx" pod "666f8f93-509b-42a4-becd-e3b89ebe144b" (UID: "666f8f93-509b-42a4-becd-e3b89ebe144b"). InnerVolumeSpecName "kube-api-access-z5gxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:18.298470 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:18.298431    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5gxx\" (UniqueName: \"kubernetes.io/projected/666f8f93-509b-42a4-becd-e3b89ebe144b-kube-api-access-z5gxx\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:18.313042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:18.313007    4559 generic.go:356] "Generic (PLEG): container finished" podID="666f8f93-509b-42a4-becd-e3b89ebe144b" containerID="6d88454b805507be2fa6bcea0da8b6a38f6d734d74f92570462307e98dc19683" exitCode=0
Oct 29 22:40:18.313221 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:18.313054    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"
Oct 29 22:40:18.313221 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:18.313089    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st" event={"ID":"666f8f93-509b-42a4-becd-e3b89ebe144b","Type":"ContainerDied","Data":"6d88454b805507be2fa6bcea0da8b6a38f6d734d74f92570462307e98dc19683"}
Oct 29 22:40:18.313221 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:18.313145    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st" event={"ID":"666f8f93-509b-42a4-becd-e3b89ebe144b","Type":"ContainerDied","Data":"f59ca4b28665a3a0bb9cc50d0ee695cd13ca13d19d205352cdd7340606dd8d0a"}
Oct 29 22:40:18.313221 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:18.313170    4559 scope.go:122] "RemoveContainer" containerID="6d88454b805507be2fa6bcea0da8b6a38f6d734d74f92570462307e98dc19683"
Oct 29 22:40:18.324546 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:18.324527    4559 scope.go:122] "RemoveContainer" containerID="6d88454b805507be2fa6bcea0da8b6a38f6d734d74f92570462307e98dc19683"
Oct 29 22:40:18.324863 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:40:18.324843    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d88454b805507be2fa6bcea0da8b6a38f6d734d74f92570462307e98dc19683\": container with ID starting with 6d88454b805507be2fa6bcea0da8b6a38f6d734d74f92570462307e98dc19683 not found: ID does not exist" containerID="6d88454b805507be2fa6bcea0da8b6a38f6d734d74f92570462307e98dc19683"
Oct 29 22:40:18.324937 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:18.324871    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d88454b805507be2fa6bcea0da8b6a38f6d734d74f92570462307e98dc19683"} err="failed to get container status \"6d88454b805507be2fa6bcea0da8b6a38f6d734d74f92570462307e98dc19683\": rpc error: code = NotFound desc = could not find container \"6d88454b805507be2fa6bcea0da8b6a38f6d734d74f92570462307e98dc19683\": container with ID starting with 6d88454b805507be2fa6bcea0da8b6a38f6d734d74f92570462307e98dc19683 not found: ID does not exist"
Oct 29 22:40:18.371232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:18.371199    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"]
Oct 29 22:40:18.377926 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:18.377894    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aeda8c-75cdbfc96-xx8st"]
Oct 29 22:40:19.325778 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:19.325711    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="666f8f93-509b-42a4-becd-e3b89ebe144b" path="/var/lib/kubelet/pods/666f8f93-509b-42a4-becd-e3b89ebe144b/volumes"
Oct 29 22:40:19.327469 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:19.327412    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb" path="/var/lib/kubelet/pods/ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb/volumes"
Oct 29 22:40:19.918956 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:19.918910    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"]
Oct 29 22:40:19.957851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:19.957781    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"]
Oct 29 22:40:20.749837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:20.748861    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"]
Oct 29 22:40:20.828868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:20.826155    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"]
Oct 29 22:40:22.739429 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:22.735092    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"]
Oct 29 22:40:24.920316 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:24.920265    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276" podUID="6a63fc53-8f88-49ef-82ae-acde0f36fd5d" containerName="payload-processing" containerID="cri-o://17e9c38421f411d1248ca8fb55016f52838466cb65ef8c3cbb1fe14901ba8f63" gracePeriod=25
Oct 29 22:40:24.959004 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:24.958952    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q" podUID="b258ed23-5da1-4feb-9b5f-a7de1f12d8b0" containerName="payload-pre-processing" containerID="cri-o://473e0d41728af5cae3d802ee755e9dd38fac8c0e4cbf8c38cc8bca3a1f15d6c8" gracePeriod=25
Oct 29 22:40:25.285520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.285485    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"
Oct 29 22:40:25.322588 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.322564    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"
Oct 29 22:40:25.400300 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.400269    4559 generic.go:356] "Generic (PLEG): container finished" podID="b258ed23-5da1-4feb-9b5f-a7de1f12d8b0" containerID="473e0d41728af5cae3d802ee755e9dd38fac8c0e4cbf8c38cc8bca3a1f15d6c8" exitCode=0
Oct 29 22:40:25.400487 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.400317    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"
Oct 29 22:40:25.400487 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.400366    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q" event={"ID":"b258ed23-5da1-4feb-9b5f-a7de1f12d8b0","Type":"ContainerDied","Data":"473e0d41728af5cae3d802ee755e9dd38fac8c0e4cbf8c38cc8bca3a1f15d6c8"}
Oct 29 22:40:25.400487 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.400420    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q" event={"ID":"b258ed23-5da1-4feb-9b5f-a7de1f12d8b0","Type":"ContainerDied","Data":"0885d6925360da9d6b85a01930f7a310e66f2830d61bdfd17d472e7c0834f5d0"}
Oct 29 22:40:25.400487 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.400450    4559 scope.go:122] "RemoveContainer" containerID="473e0d41728af5cae3d802ee755e9dd38fac8c0e4cbf8c38cc8bca3a1f15d6c8"
Oct 29 22:40:25.402345 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.402316    4559 generic.go:356] "Generic (PLEG): container finished" podID="6a63fc53-8f88-49ef-82ae-acde0f36fd5d" containerID="17e9c38421f411d1248ca8fb55016f52838466cb65ef8c3cbb1fe14901ba8f63" exitCode=0
Oct 29 22:40:25.402452 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.402362    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"
Oct 29 22:40:25.402452 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.402361    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276" event={"ID":"6a63fc53-8f88-49ef-82ae-acde0f36fd5d","Type":"ContainerDied","Data":"17e9c38421f411d1248ca8fb55016f52838466cb65ef8c3cbb1fe14901ba8f63"}
Oct 29 22:40:25.402452 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.402419    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276" event={"ID":"6a63fc53-8f88-49ef-82ae-acde0f36fd5d","Type":"ContainerDied","Data":"c4525583becbcb9da5d2963c3c651a7959bd16b9c26c9c69ea3465344635e399"}
Oct 29 22:40:25.411820 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.411789    4559 scope.go:122] "RemoveContainer" containerID="473e0d41728af5cae3d802ee755e9dd38fac8c0e4cbf8c38cc8bca3a1f15d6c8"
Oct 29 22:40:25.412145 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:40:25.412120    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"473e0d41728af5cae3d802ee755e9dd38fac8c0e4cbf8c38cc8bca3a1f15d6c8\": container with ID starting with 473e0d41728af5cae3d802ee755e9dd38fac8c0e4cbf8c38cc8bca3a1f15d6c8 not found: ID does not exist" containerID="473e0d41728af5cae3d802ee755e9dd38fac8c0e4cbf8c38cc8bca3a1f15d6c8"
Oct 29 22:40:25.412258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.412150    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"473e0d41728af5cae3d802ee755e9dd38fac8c0e4cbf8c38cc8bca3a1f15d6c8"} err="failed to get container status \"473e0d41728af5cae3d802ee755e9dd38fac8c0e4cbf8c38cc8bca3a1f15d6c8\": rpc error: code = NotFound desc = could not find container \"473e0d41728af5cae3d802ee755e9dd38fac8c0e4cbf8c38cc8bca3a1f15d6c8\": container with ID starting with 473e0d41728af5cae3d802ee755e9dd38fac8c0e4cbf8c38cc8bca3a1f15d6c8 not found: ID does not exist"
Oct 29 22:40:25.412258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.412168    4559 scope.go:122] "RemoveContainer" containerID="17e9c38421f411d1248ca8fb55016f52838466cb65ef8c3cbb1fe14901ba8f63"
Oct 29 22:40:25.422620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.422603    4559 scope.go:122] "RemoveContainer" containerID="17e9c38421f411d1248ca8fb55016f52838466cb65ef8c3cbb1fe14901ba8f63"
Oct 29 22:40:25.422966 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:40:25.422946    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e9c38421f411d1248ca8fb55016f52838466cb65ef8c3cbb1fe14901ba8f63\": container with ID starting with 17e9c38421f411d1248ca8fb55016f52838466cb65ef8c3cbb1fe14901ba8f63 not found: ID does not exist" containerID="17e9c38421f411d1248ca8fb55016f52838466cb65ef8c3cbb1fe14901ba8f63"
Oct 29 22:40:25.423028 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.422972    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e9c38421f411d1248ca8fb55016f52838466cb65ef8c3cbb1fe14901ba8f63"} err="failed to get container status \"17e9c38421f411d1248ca8fb55016f52838466cb65ef8c3cbb1fe14901ba8f63\": rpc error: code = NotFound desc = could not find container \"17e9c38421f411d1248ca8fb55016f52838466cb65ef8c3cbb1fe14901ba8f63\": container with ID starting with 17e9c38421f411d1248ca8fb55016f52838466cb65ef8c3cbb1fe14901ba8f63 not found: ID does not exist"
Oct 29 22:40:25.433005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.432973    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b258ed23-5da1-4feb-9b5f-a7de1f12d8b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b258ed23-5da1-4feb-9b5f-a7de1f12d8b0-plugins-config-volume\") pod \"b258ed23-5da1-4feb-9b5f-a7de1f12d8b0\" (UID: \"b258ed23-5da1-4feb-9b5f-a7de1f12d8b0\") "
Oct 29 22:40:25.433145 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.433082    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b258ed23-5da1-4feb-9b5f-a7de1f12d8b0-kube-api-access-l54jm\" (UniqueName: \"kubernetes.io/projected/b258ed23-5da1-4feb-9b5f-a7de1f12d8b0-kube-api-access-l54jm\") pod \"b258ed23-5da1-4feb-9b5f-a7de1f12d8b0\" (UID: \"b258ed23-5da1-4feb-9b5f-a7de1f12d8b0\") "
Oct 29 22:40:25.433145 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.433126    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a63fc53-8f88-49ef-82ae-acde0f36fd5d-kube-api-access-75chl\" (UniqueName: \"kubernetes.io/projected/6a63fc53-8f88-49ef-82ae-acde0f36fd5d-kube-api-access-75chl\") pod \"6a63fc53-8f88-49ef-82ae-acde0f36fd5d\" (UID: \"6a63fc53-8f88-49ef-82ae-acde0f36fd5d\") "
Oct 29 22:40:25.433271 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.433243    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a63fc53-8f88-49ef-82ae-acde0f36fd5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a63fc53-8f88-49ef-82ae-acde0f36fd5d-plugins-config-volume\") pod \"6a63fc53-8f88-49ef-82ae-acde0f36fd5d\" (UID: \"6a63fc53-8f88-49ef-82ae-acde0f36fd5d\") "
Oct 29 22:40:25.433399 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.433372    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b258ed23-5da1-4feb-9b5f-a7de1f12d8b0-plugins-config-volume" pod "b258ed23-5da1-4feb-9b5f-a7de1f12d8b0" (UID: "b258ed23-5da1-4feb-9b5f-a7de1f12d8b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:25.434244 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.434217    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b258ed23-5da1-4feb-9b5f-a7de1f12d8b0-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:25.434356 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.434334    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a63fc53-8f88-49ef-82ae-acde0f36fd5d-plugins-config-volume" pod "6a63fc53-8f88-49ef-82ae-acde0f36fd5d" (UID: "6a63fc53-8f88-49ef-82ae-acde0f36fd5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:25.435994 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.435967    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b258ed23-5da1-4feb-9b5f-a7de1f12d8b0-kube-api-access-l54jm" pod "b258ed23-5da1-4feb-9b5f-a7de1f12d8b0" (UID: "b258ed23-5da1-4feb-9b5f-a7de1f12d8b0"). InnerVolumeSpecName "kube-api-access-l54jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:25.436521 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.436500    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a63fc53-8f88-49ef-82ae-acde0f36fd5d-kube-api-access-75chl" pod "6a63fc53-8f88-49ef-82ae-acde0f36fd5d" (UID: "6a63fc53-8f88-49ef-82ae-acde0f36fd5d"). InnerVolumeSpecName "kube-api-access-75chl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:25.536023 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.535924    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l54jm\" (UniqueName: \"kubernetes.io/projected/b258ed23-5da1-4feb-9b5f-a7de1f12d8b0-kube-api-access-l54jm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:25.536023 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.535960    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75chl\" (UniqueName: \"kubernetes.io/projected/6a63fc53-8f88-49ef-82ae-acde0f36fd5d-kube-api-access-75chl\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:25.536023 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.535973    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a63fc53-8f88-49ef-82ae-acde0f36fd5d-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:25.748753 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.748712    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"]
Oct 29 22:40:25.750096 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.750027    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d" podUID="fff6a521-a276-49f3-a64d-5929e6cc963d" containerName="payload-processing" containerID="cri-o://4d905f4eefec9f8863d2f20c7c9030a4a56a45c7819c807b335caaba75663804" gracePeriod=25
Oct 29 22:40:25.764014 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.763966    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-39322f-6bd999dc7f-dcj8q"]
Oct 29 22:40:25.796743 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.796661    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"]
Oct 29 22:40:25.800631 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.800609    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-39322f-5c74d47cd-88276"]
Oct 29 22:40:25.827246 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.827174    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w" podUID="5271c98e-3534-4fbc-8908-4ecc13444202" containerName="payload-pre-processing" containerID="cri-o://3857ae3b65df82fbe436a830701d3095760006cde89559559703ff19d10a78d4" gracePeriod=25
Oct 29 22:40:25.902495 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.902458    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"]
Oct 29 22:40:25.903235 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903219    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb" containerName="maas-api"
Oct 29 22:40:25.903288 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903236    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb" containerName="maas-api"
Oct 29 22:40:25.903288 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903253    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerName="istio-proxy"
Oct 29 22:40:25.903288 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903258    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerName="istio-proxy"
Oct 29 22:40:25.903288 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903266    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerName="istio-proxy"
Oct 29 22:40:25.903288 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903272    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerName="istio-proxy"
Oct 29 22:40:25.903459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903291    4559 cpu_manager.go:414] "containerMap: removing container" podUID="cd055ebb-b30b-447d-831d-1ff95c57dc55" containerName="istio-proxy"
Oct 29 22:40:25.903459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903296    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd055ebb-b30b-447d-831d-1ff95c57dc55" containerName="istio-proxy"
Oct 29 22:40:25.903459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903313    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerName="istio-proxy"
Oct 29 22:40:25.903459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903318    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerName="istio-proxy"
Oct 29 22:40:25.903459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903331    4559 cpu_manager.go:414] "containerMap: removing container" podUID="666f8f93-509b-42a4-becd-e3b89ebe144b" containerName="payload-processing"
Oct 29 22:40:25.903459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903335    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="666f8f93-509b-42a4-becd-e3b89ebe144b" containerName="payload-processing"
Oct 29 22:40:25.903459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903345    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b258ed23-5da1-4feb-9b5f-a7de1f12d8b0" containerName="payload-pre-processing"
Oct 29 22:40:25.903459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903349    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b258ed23-5da1-4feb-9b5f-a7de1f12d8b0" containerName="payload-pre-processing"
Oct 29 22:40:25.903459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903363    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6a63fc53-8f88-49ef-82ae-acde0f36fd5d" containerName="payload-processing"
Oct 29 22:40:25.903459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903368    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a63fc53-8f88-49ef-82ae-acde0f36fd5d" containerName="payload-processing"
Oct 29 22:40:25.903759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903464    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerName="istio-proxy"
Oct 29 22:40:25.903759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903473    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="666f8f93-509b-42a4-becd-e3b89ebe144b" containerName="payload-processing"
Oct 29 22:40:25.903759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903480    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b258ed23-5da1-4feb-9b5f-a7de1f12d8b0" containerName="payload-pre-processing"
Oct 29 22:40:25.903759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903489    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a63fc53-8f88-49ef-82ae-acde0f36fd5d" containerName="payload-processing"
Oct 29 22:40:25.903759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903497    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4e3a3a-c3fa-4dcd-aa66-f5d7158bfedb" containerName="maas-api"
Oct 29 22:40:25.903759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903505    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerName="istio-proxy"
Oct 29 22:40:25.903759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903521    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd055ebb-b30b-447d-831d-1ff95c57dc55" containerName="istio-proxy"
Oct 29 22:40:25.903759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.903530    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2040bbf0-5596-401b-b7f6-b3fa571c4d94" containerName="istio-proxy"
Oct 29 22:40:25.907220 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.907201    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:40:25.926518 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.926009    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8d985a-serving-cert\""
Oct 29 22:40:25.946302 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:25.931684    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"]
Oct 29 22:40:26.047021 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.045466    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2794j\" (UniqueName: \"kubernetes.io/projected/38c2e753-0737-4b90-a553-428308a0d2ab-kube-api-access-2794j\") pod \"maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj\" (UID: \"38c2e753-0737-4b90-a553-428308a0d2ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:40:26.047021 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.045525    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c2e753-0737-4b90-a553-428308a0d2ab-metrics-certs\") pod \"maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj\" (UID: \"38c2e753-0737-4b90-a553-428308a0d2ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:40:26.047021 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.045954    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c2e753-0737-4b90-a553-428308a0d2ab-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj\" (UID: \"38c2e753-0737-4b90-a553-428308a0d2ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:40:26.047021 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.045993    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c2e753-0737-4b90-a553-428308a0d2ab-maas-api-tls\") pod \"maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj\" (UID: \"38c2e753-0737-4b90-a553-428308a0d2ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:40:26.202841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.193531    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c2e753-0737-4b90-a553-428308a0d2ab-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj\" (UID: \"38c2e753-0737-4b90-a553-428308a0d2ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:40:26.228851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.213289    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c2e753-0737-4b90-a553-428308a0d2ab-maas-api-tls\") pod \"maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj\" (UID: \"38c2e753-0737-4b90-a553-428308a0d2ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:40:26.228851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.213462    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2794j\" (UniqueName: \"kubernetes.io/projected/38c2e753-0737-4b90-a553-428308a0d2ab-kube-api-access-2794j\") pod \"maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj\" (UID: \"38c2e753-0737-4b90-a553-428308a0d2ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:40:26.228851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.213515    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c2e753-0737-4b90-a553-428308a0d2ab-metrics-certs\") pod \"maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj\" (UID: \"38c2e753-0737-4b90-a553-428308a0d2ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:40:26.275907 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:40:26.254352    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-serving-cert: secret "maas-api-e2e-inf-b-8d985a-serving-cert" not found
Oct 29 22:40:26.275907 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:40:26.254430    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c2e753-0737-4b90-a553-428308a0d2ab-maas-api-tls podName:38c2e753-0737-4b90-a553-428308a0d2ab nodeName:}" failed. No retries permitted until 2026-10-29 22:40:26.754409276 +0000 UTC m=+1565.665852825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/38c2e753-0737-4b90-a553-428308a0d2ab-maas-api-tls") pod "maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj" (UID: "38c2e753-0737-4b90-a553-428308a0d2ab") : secret "maas-api-e2e-inf-b-8d985a-serving-cert" not found
Oct 29 22:40:26.303591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.303504    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c2e753-0737-4b90-a553-428308a0d2ab-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj\" (UID: \"38c2e753-0737-4b90-a553-428308a0d2ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:40:26.312891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.312742    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2794j\" (UniqueName: \"kubernetes.io/projected/38c2e753-0737-4b90-a553-428308a0d2ab-kube-api-access-2794j\") pod \"maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj\" (UID: \"38c2e753-0737-4b90-a553-428308a0d2ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:40:26.316422 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.316091    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c2e753-0737-4b90-a553-428308a0d2ab-metrics-certs\") pod \"maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj\" (UID: \"38c2e753-0737-4b90-a553-428308a0d2ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:40:26.363367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.363331    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"
Oct 29 22:40:26.416043 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.415917    4559 generic.go:356] "Generic (PLEG): container finished" podID="fff6a521-a276-49f3-a64d-5929e6cc963d" containerID="4d905f4eefec9f8863d2f20c7c9030a4a56a45c7819c807b335caaba75663804" exitCode=0
Oct 29 22:40:26.416043 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.416021    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d" event={"ID":"fff6a521-a276-49f3-a64d-5929e6cc963d","Type":"ContainerDied","Data":"4d905f4eefec9f8863d2f20c7c9030a4a56a45c7819c807b335caaba75663804"}
Oct 29 22:40:26.416641 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.416478    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5271c98e-3534-4fbc-8908-4ecc13444202-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5271c98e-3534-4fbc-8908-4ecc13444202-plugins-config-volume\") pod \"5271c98e-3534-4fbc-8908-4ecc13444202\" (UID: \"5271c98e-3534-4fbc-8908-4ecc13444202\") "
Oct 29 22:40:26.416641 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.416583    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5271c98e-3534-4fbc-8908-4ecc13444202-kube-api-access-bh62j\" (UniqueName: \"kubernetes.io/projected/5271c98e-3534-4fbc-8908-4ecc13444202-kube-api-access-bh62j\") pod \"5271c98e-3534-4fbc-8908-4ecc13444202\" (UID: \"5271c98e-3534-4fbc-8908-4ecc13444202\") "
Oct 29 22:40:26.417138 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.417109    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5271c98e-3534-4fbc-8908-4ecc13444202-plugins-config-volume" pod "5271c98e-3534-4fbc-8908-4ecc13444202" (UID: "5271c98e-3534-4fbc-8908-4ecc13444202"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:26.417300 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.417279    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5271c98e-3534-4fbc-8908-4ecc13444202-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:26.419618 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.419590    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5271c98e-3534-4fbc-8908-4ecc13444202-kube-api-access-bh62j" pod "5271c98e-3534-4fbc-8908-4ecc13444202" (UID: "5271c98e-3534-4fbc-8908-4ecc13444202"). InnerVolumeSpecName "kube-api-access-bh62j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:26.420777 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.420755    4559 generic.go:356] "Generic (PLEG): container finished" podID="5271c98e-3534-4fbc-8908-4ecc13444202" containerID="3857ae3b65df82fbe436a830701d3095760006cde89559559703ff19d10a78d4" exitCode=0
Oct 29 22:40:26.420925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.420856    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w" event={"ID":"5271c98e-3534-4fbc-8908-4ecc13444202","Type":"ContainerDied","Data":"3857ae3b65df82fbe436a830701d3095760006cde89559559703ff19d10a78d4"}
Oct 29 22:40:26.420925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.420895    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w" event={"ID":"5271c98e-3534-4fbc-8908-4ecc13444202","Type":"ContainerDied","Data":"072937ca45aaf7971025ab6ffa8f981c863a1039292d56a750cba9d3aa972d5b"}
Oct 29 22:40:26.420925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.420918    4559 scope.go:122] "RemoveContainer" containerID="3857ae3b65df82fbe436a830701d3095760006cde89559559703ff19d10a78d4"
Oct 29 22:40:26.421098 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.421024    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"
Oct 29 22:40:26.468756 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.468732    4559 scope.go:122] "RemoveContainer" containerID="3857ae3b65df82fbe436a830701d3095760006cde89559559703ff19d10a78d4"
Oct 29 22:40:26.469424 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:40:26.469399    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3857ae3b65df82fbe436a830701d3095760006cde89559559703ff19d10a78d4\": container with ID starting with 3857ae3b65df82fbe436a830701d3095760006cde89559559703ff19d10a78d4 not found: ID does not exist" containerID="3857ae3b65df82fbe436a830701d3095760006cde89559559703ff19d10a78d4"
Oct 29 22:40:26.469539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.469433    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3857ae3b65df82fbe436a830701d3095760006cde89559559703ff19d10a78d4"} err="failed to get container status \"3857ae3b65df82fbe436a830701d3095760006cde89559559703ff19d10a78d4\": rpc error: code = NotFound desc = could not find container \"3857ae3b65df82fbe436a830701d3095760006cde89559559703ff19d10a78d4\": container with ID starting with 3857ae3b65df82fbe436a830701d3095760006cde89559559703ff19d10a78d4 not found: ID does not exist"
Oct 29 22:40:26.474421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.474402    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"
Oct 29 22:40:26.508112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.508081    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"]
Oct 29 22:40:26.513550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.513518    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3d1777-5d6bcd9959-52r9w"]
Oct 29 22:40:26.517995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.517967    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff6a521-a276-49f3-a64d-5929e6cc963d-kube-api-access-z9kqc\" (UniqueName: \"kubernetes.io/projected/fff6a521-a276-49f3-a64d-5929e6cc963d-kube-api-access-z9kqc\") pod \"fff6a521-a276-49f3-a64d-5929e6cc963d\" (UID: \"fff6a521-a276-49f3-a64d-5929e6cc963d\") "
Oct 29 22:40:26.518123 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.518094    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff6a521-a276-49f3-a64d-5929e6cc963d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fff6a521-a276-49f3-a64d-5929e6cc963d-plugins-config-volume\") pod \"fff6a521-a276-49f3-a64d-5929e6cc963d\" (UID: \"fff6a521-a276-49f3-a64d-5929e6cc963d\") "
Oct 29 22:40:26.518455 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.518433    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff6a521-a276-49f3-a64d-5929e6cc963d-plugins-config-volume" pod "fff6a521-a276-49f3-a64d-5929e6cc963d" (UID: "fff6a521-a276-49f3-a64d-5929e6cc963d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:26.518757 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.518735    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh62j\" (UniqueName: \"kubernetes.io/projected/5271c98e-3534-4fbc-8908-4ecc13444202-kube-api-access-bh62j\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:26.518757 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.518752    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fff6a521-a276-49f3-a64d-5929e6cc963d-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:26.520245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.520225    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff6a521-a276-49f3-a64d-5929e6cc963d-kube-api-access-z9kqc" pod "fff6a521-a276-49f3-a64d-5929e6cc963d" (UID: "fff6a521-a276-49f3-a64d-5929e6cc963d"). InnerVolumeSpecName "kube-api-access-z9kqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:26.620602 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.620518    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9kqc\" (UniqueName: \"kubernetes.io/projected/fff6a521-a276-49f3-a64d-5929e6cc963d-kube-api-access-z9kqc\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:26.828890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.828850    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c2e753-0737-4b90-a553-428308a0d2ab-maas-api-tls\") pod \"maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj\" (UID: \"38c2e753-0737-4b90-a553-428308a0d2ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:40:26.831464 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:26.831435    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c2e753-0737-4b90-a553-428308a0d2ab-maas-api-tls\") pod \"maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj\" (UID: \"38c2e753-0737-4b90-a553-428308a0d2ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:40:27.133838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.129009    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:40:27.305362 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.305323    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5271c98e-3534-4fbc-8908-4ecc13444202" path="/var/lib/kubelet/pods/5271c98e-3534-4fbc-8908-4ecc13444202/volumes"
Oct 29 22:40:27.306337 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.306312    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a63fc53-8f88-49ef-82ae-acde0f36fd5d" path="/var/lib/kubelet/pods/6a63fc53-8f88-49ef-82ae-acde0f36fd5d/volumes"
Oct 29 22:40:27.307103 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.307085    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b258ed23-5da1-4feb-9b5f-a7de1f12d8b0" path="/var/lib/kubelet/pods/b258ed23-5da1-4feb-9b5f-a7de1f12d8b0/volumes"
Oct 29 22:40:27.474946 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.472625    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d" event={"ID":"fff6a521-a276-49f3-a64d-5929e6cc963d","Type":"ContainerDied","Data":"3790781ef2650e0baed0aed7120b0a7cc037e6954aab1794a2a3f4ecd5d2af53"}
Oct 29 22:40:27.474946 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.472696    4559 scope.go:122] "RemoveContainer" containerID="4d905f4eefec9f8863d2f20c7c9030a4a56a45c7819c807b335caaba75663804"
Oct 29 22:40:27.474946 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.472829    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"
Oct 29 22:40:27.536293 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.536104    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"]
Oct 29 22:40:27.554244 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.554214    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d1777-6dfd6d6694-vmg4d"]
Oct 29 22:40:27.574437 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.574299    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"]
Oct 29 22:40:27.575562 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.575519    4559 cpu_manager.go:414] "containerMap: removing container" podUID="5271c98e-3534-4fbc-8908-4ecc13444202" containerName="payload-pre-processing"
Oct 29 22:40:27.575562 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.575548    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5271c98e-3534-4fbc-8908-4ecc13444202" containerName="payload-pre-processing"
Oct 29 22:40:27.575861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.575620    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerName="istio-proxy"
Oct 29 22:40:27.575861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.575629    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerName="istio-proxy"
Oct 29 22:40:27.575861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.575644    4559 cpu_manager.go:414] "containerMap: removing container" podUID="fff6a521-a276-49f3-a64d-5929e6cc963d" containerName="payload-processing"
Oct 29 22:40:27.575861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.575649    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff6a521-a276-49f3-a64d-5929e6cc963d" containerName="payload-processing"
Oct 29 22:40:27.575861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.575776    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff6a521-a276-49f3-a64d-5929e6cc963d" containerName="payload-processing"
Oct 29 22:40:27.575861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.575793    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5762b8-dbb2-4967-a9c5-8a2876e913fb" containerName="istio-proxy"
Oct 29 22:40:27.575861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.575801    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="5271c98e-3534-4fbc-8908-4ecc13444202" containerName="payload-pre-processing"
Oct 29 22:40:27.581389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.581364    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.601529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.601500    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"]
Oct 29 22:40:27.647673 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.647634    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"]
Oct 29 22:40:27.649006 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:40:27.648960    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c2e753_0737_4b90_a553_428308a0d2ab.slice/crio-c48d1417499ecac8ef250a54513fd12425dd1f9931e2f6d3f6f7ed7c2eb8fae0 WatchSource:0}: Error finding container c48d1417499ecac8ef250a54513fd12425dd1f9931e2f6d3f6f7ed7c2eb8fae0: Status 404 returned error can't find the container with id c48d1417499ecac8ef250a54513fd12425dd1f9931e2f6d3f6f7ed7c2eb8fae0
Oct 29 22:40:27.683236 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.683192    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.683449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.683282    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.683449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.683330    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22tlv\" (UniqueName: \"kubernetes.io/projected/724aa157-d061-4f4d-8bcd-72d474e7207d-kube-api-access-22tlv\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.683449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.683412    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.683449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.683438    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/724aa157-d061-4f4d-8bcd-72d474e7207d-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.683699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.683460    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.683699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.683625    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.683828 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.683744    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.683828 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.683770    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.795890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.792109    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.795890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.792172    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.795890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.792299    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.795890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.792351    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.795890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.792398    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22tlv\" (UniqueName: \"kubernetes.io/projected/724aa157-d061-4f4d-8bcd-72d474e7207d-kube-api-access-22tlv\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.795890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.792478    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.795890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.792504    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/724aa157-d061-4f4d-8bcd-72d474e7207d-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.795890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.792528    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.795890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.792692    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.795890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.793347    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.817836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.817488    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.817836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.817748    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.824834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.818377    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.824834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.819105    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/724aa157-d061-4f4d-8bcd-72d474e7207d-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.824834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.819637    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.836878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.833268    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.870600 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.864442    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.870600 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.865228    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22tlv\" (UniqueName: \"kubernetes.io/projected/724aa157-d061-4f4d-8bcd-72d474e7207d-kube-api-access-22tlv\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:27.901273 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:27.901236    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:28.245022 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:28.244980    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"]
Oct 29 22:40:28.255960 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:40:28.255930    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724aa157_d061_4f4d_8bcd_72d474e7207d.slice/crio-a33801e220ff0a2cf252d07644dc61ae04d191484da9bcfc1819b0165efbd970 WatchSource:0}: Error finding container a33801e220ff0a2cf252d07644dc61ae04d191484da9bcfc1819b0165efbd970: Status 404 returned error can't find the container with id a33801e220ff0a2cf252d07644dc61ae04d191484da9bcfc1819b0165efbd970
Oct 29 22:40:28.259409 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:28.259364    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:40:28.259559 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:28.259466    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:40:28.259559 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:28.259509    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:40:28.485314 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:28.485278    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj" event={"ID":"38c2e753-0737-4b90-a553-428308a0d2ab","Type":"ContainerStarted","Data":"969b58ad05620973e6dfbbb71e363e5dc9dd1a69c1e865b0dc79ae7b133dee16"}
Oct 29 22:40:28.485573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:28.485337    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj" event={"ID":"38c2e753-0737-4b90-a553-428308a0d2ab","Type":"ContainerStarted","Data":"c48d1417499ecac8ef250a54513fd12425dd1f9931e2f6d3f6f7ed7c2eb8fae0"}
Oct 29 22:40:28.485573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:28.485486    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:40:28.489221 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:28.489184    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4" event={"ID":"724aa157-d061-4f4d-8bcd-72d474e7207d","Type":"ContainerStarted","Data":"65879f6d2226944f10890f9ed3492b76361708fcb579eb30dec0e317e97d5d90"}
Oct 29 22:40:28.489396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:28.489247    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4" event={"ID":"724aa157-d061-4f4d-8bcd-72d474e7207d","Type":"ContainerStarted","Data":"a33801e220ff0a2cf252d07644dc61ae04d191484da9bcfc1819b0165efbd970"}
Oct 29 22:40:28.530169 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:28.530109    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj" podStartSLOduration=3.109885911 podStartE2EDuration="3.530092622s" totalImagesPullingTime="420.206711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:28.525695962 +0000 UTC m=+1567.437139519" watchObservedRunningTime="2026-10-29 22:40:28.530092622 +0000 UTC m=+1567.441536175"
Oct 29 22:40:28.581853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:28.581706    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4" podStartSLOduration=1.5816834119999998 podStartE2EDuration="1.581683412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:28.580336133 +0000 UTC m=+1567.491779690" watchObservedRunningTime="2026-10-29 22:40:28.581683412 +0000 UTC m=+1567.493126973"
Oct 29 22:40:28.907265 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:28.907226    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:29.229843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:29.229747    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff6a521-a276-49f3-a64d-5929e6cc963d" path="/var/lib/kubelet/pods/fff6a521-a276-49f3-a64d-5929e6cc963d/volumes"
Oct 29 22:40:29.904177 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:29.904143    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:40:29.904608 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:29.904200    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4" podUID="724aa157-d061-4f4d-8bcd-72d474e7207d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 29 22:40:30.902500 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:30.902464    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:40:30.902692 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:30.902517    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4" podUID="724aa157-d061-4f4d-8bcd-72d474e7207d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 29 22:40:31.902463 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:31.902428    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:40:31.902918 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:31.902478    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4" podUID="724aa157-d061-4f4d-8bcd-72d474e7207d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 29 22:40:32.902921 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:32.902785    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:40:32.902921 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:32.902896    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4" podUID="724aa157-d061-4f4d-8bcd-72d474e7207d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 29 22:40:33.910641 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:33.909911    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:40:33.911270 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:33.911227    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4" podUID="724aa157-d061-4f4d-8bcd-72d474e7207d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 29 22:40:34.558512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:34.558474    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:40:34.906270 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:34.906235    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:40:34.906587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:34.906554    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4" podUID="724aa157-d061-4f4d-8bcd-72d474e7207d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 29 22:40:35.907650 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:35.907610    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:35.908131 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:35.908046    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:35.909065 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:35.909039    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:40:37.766557 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:37.766503    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main" containerID="cri-o://8b0d6b0b531785984d32f9bad16c126d31c415e5148631ea31bd92023390a36a" gracePeriod=45
Oct 29 22:40:38.075503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.075476    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:40:38.275834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.266361    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-tls-certs\" (UniqueName: \"kubernetes.io/secret/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-tls-certs\") pod \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") "
Oct 29 22:40:38.275834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.266467    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-tmp-dir\") pod \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") "
Oct 29 22:40:38.275834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.266711    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-kube-api-access-bqb6z\" (UniqueName: \"kubernetes.io/projected/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-kube-api-access-bqb6z\") pod \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") "
Oct 29 22:40:38.275834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.266780    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-model-cache\") pod \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") "
Oct 29 22:40:38.298333 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.297957    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-home\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-home\") pod \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") "
Oct 29 22:40:38.298333 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.298046    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-dshm\") pod \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\" (UID: \"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73\") "
Oct 29 22:40:38.307237 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.306838    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-dshm" pod "f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" (UID: "f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:38.317834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.316263    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-model-cache" pod "f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" (UID: "f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:38.317834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.316472    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-tmp-dir" pod "f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" (UID: "f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:38.317834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.316639    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-home" pod "f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" (UID: "f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:38.322300 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.320305    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-kube-api-access-bqb6z" pod "f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" (UID: "f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73"). InnerVolumeSpecName "kube-api-access-bqb6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:38.322914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.322569    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-tls-certs" pod "f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" (UID: "f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:40:38.401411 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.401378    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:38.401411 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.401407    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:38.401639 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.401420    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqb6z\" (UniqueName: \"kubernetes.io/projected/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-kube-api-access-bqb6z\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:38.401639 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.401435    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:38.401639 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.401446    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:38.401639 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.401456    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:38.750967 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.750923    4559 generic.go:356] "Generic (PLEG): container finished" podID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerID="8b0d6b0b531785984d32f9bad16c126d31c415e5148631ea31bd92023390a36a" exitCode=0
Oct 29 22:40:38.751188 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.750976    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"
Oct 29 22:40:38.751188 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.751002    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" event={"ID":"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73","Type":"ContainerDied","Data":"8b0d6b0b531785984d32f9bad16c126d31c415e5148631ea31bd92023390a36a"}
Oct 29 22:40:38.751188 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.751051    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8" event={"ID":"f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73","Type":"ContainerDied","Data":"23700c9e37793af0793db7a1fc016c4c460483fe6059d7facb5144c5aea0edef"}
Oct 29 22:40:38.751188 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.751070    4559 scope.go:122] "RemoveContainer" containerID="8b0d6b0b531785984d32f9bad16c126d31c415e5148631ea31bd92023390a36a"
Oct 29 22:40:38.762260 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.762235    4559 scope.go:122] "RemoveContainer" containerID="8b0d6b0b531785984d32f9bad16c126d31c415e5148631ea31bd92023390a36a"
Oct 29 22:40:38.762537 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:40:38.762518    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0d6b0b531785984d32f9bad16c126d31c415e5148631ea31bd92023390a36a\": container with ID starting with 8b0d6b0b531785984d32f9bad16c126d31c415e5148631ea31bd92023390a36a not found: ID does not exist" containerID="8b0d6b0b531785984d32f9bad16c126d31c415e5148631ea31bd92023390a36a"
Oct 29 22:40:38.762603 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.762546    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0d6b0b531785984d32f9bad16c126d31c415e5148631ea31bd92023390a36a"} err="failed to get container status \"8b0d6b0b531785984d32f9bad16c126d31c415e5148631ea31bd92023390a36a\": rpc error: code = NotFound desc = could not find container \"8b0d6b0b531785984d32f9bad16c126d31c415e5148631ea31bd92023390a36a\": container with ID starting with 8b0d6b0b531785984d32f9bad16c126d31c415e5148631ea31bd92023390a36a not found: ID does not exist"
Oct 29 22:40:38.807032 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.806999    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"]
Oct 29 22:40:38.819417 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:38.819384    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-1fa4b8b6/e2e-lifecycle-model-1fa4b8b6-kserve-6bc59796f-d9ft8"]
Oct 29 22:40:39.230564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:39.230525    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" path="/var/lib/kubelet/pods/f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73/volumes"
Oct 29 22:40:42.618550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.618509    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"]
Oct 29 22:40:42.619407 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.619382    4559 cpu_manager.go:414] "containerMap: removing container" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main"
Oct 29 22:40:42.619407 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.619404    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main"
Oct 29 22:40:42.619644 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.619582    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31c5a3f-ff14-4fb7-b42c-ed9c9d2c2a73" containerName="main"
Oct 29 22:40:42.623313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.623291    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.656381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.656347    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"]
Oct 29 22:40:42.675002 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.674963    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.675385 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.675358    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d0c0ae-111d-4f92-ac13-58201176aa4c-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.675647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.675626    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.675868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.675847    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpg2b\" (UniqueName: \"kubernetes.io/projected/18d0c0ae-111d-4f92-ac13-58201176aa4c-kube-api-access-vpg2b\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.676085 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.676067    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.676340 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.676321    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.676542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.676525    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.676855 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.676834    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.677086 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.677068    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.779398 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.779358    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.779629 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.779439    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.779629 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.779520    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.779629 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.779596    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.779799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.779700    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.779799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.779772    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d0c0ae-111d-4f92-ac13-58201176aa4c-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.779979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.779844    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.779979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.779884    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.779979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.779942    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpg2b\" (UniqueName: \"kubernetes.io/projected/18d0c0ae-111d-4f92-ac13-58201176aa4c-kube-api-access-vpg2b\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.780125 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.780006    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.780180 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.780127    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.780477 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.780416    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.780768 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.780745    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d0c0ae-111d-4f92-ac13-58201176aa4c-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.781676 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.781648    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.782726 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.782701    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.783033 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.783014    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.802073 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.802037    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.802246 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.802103    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpg2b\" (UniqueName: \"kubernetes.io/projected/18d0c0ae-111d-4f92-ac13-58201176aa4c-kube-api-access-vpg2b\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:42.938672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:42.938637    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:43.489011 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:40:43.488978    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d0c0ae_111d_4f92_ac13_58201176aa4c.slice/crio-ac29b6be8fe1c1b1dfda60e6ec7943d1a5ef98575883747eb7f5351f60bb0423 WatchSource:0}: Error finding container ac29b6be8fe1c1b1dfda60e6ec7943d1a5ef98575883747eb7f5351f60bb0423: Status 404 returned error can't find the container with id ac29b6be8fe1c1b1dfda60e6ec7943d1a5ef98575883747eb7f5351f60bb0423
Oct 29 22:40:43.489331 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:43.489310    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"]
Oct 29 22:40:43.491913 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:43.491884    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:40:43.492012 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:43.491995    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:40:43.492066 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:43.492045    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:40:43.700898 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:40:43.700856    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:40:43.799446 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:43.799359    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8" event={"ID":"18d0c0ae-111d-4f92-ac13-58201176aa4c","Type":"ContainerStarted","Data":"e4e583091db7b0c2965c0f888d4d74a512706f89f013bb25df98416083cb5a85"}
Oct 29 22:40:43.799446 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:43.799425    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8" event={"ID":"18d0c0ae-111d-4f92-ac13-58201176aa4c","Type":"ContainerStarted","Data":"ac29b6be8fe1c1b1dfda60e6ec7943d1a5ef98575883747eb7f5351f60bb0423"}
Oct 29 22:40:43.850460 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:43.850388    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8" podStartSLOduration=1.8503688230000002 podStartE2EDuration="1.850368823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:43.846387777 +0000 UTC m=+1582.757831330" watchObservedRunningTime="2026-10-29 22:40:43.850368823 +0000 UTC m=+1582.761812378"
Oct 29 22:40:43.939327 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:43.939289    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:44.952836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:44.947933    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 29 22:40:44.952836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:44.948003    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8" podUID="18d0c0ae-111d-4f92-ac13-58201176aa4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 29 22:40:45.941312 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:45.941270    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 29 22:40:45.941613 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:45.941581    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8" podUID="18d0c0ae-111d-4f92-ac13-58201176aa4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 29 22:40:46.940473 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:46.940430    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 29 22:40:46.941259 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:46.941221    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8" podUID="18d0c0ae-111d-4f92-ac13-58201176aa4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 29 22:40:47.957261 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:47.957224    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 29 22:40:47.957909 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:47.957874    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8" podUID="18d0c0ae-111d-4f92-ac13-58201176aa4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 29 22:40:48.958836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:48.940240    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 29 22:40:48.958836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:48.940306    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8" podUID="18d0c0ae-111d-4f92-ac13-58201176aa4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 29 22:40:49.942846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:49.942237    4559 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 29 22:40:49.942846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:49.942300    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8" podUID="18d0c0ae-111d-4f92-ac13-58201176aa4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 29 22:40:50.983925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:50.982244    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:50.984358 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:50.984015    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:50.985367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:50.985343    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:40:51.921691 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:51.921640    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"]
Oct 29 22:40:51.927677 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:51.927647    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:51.966381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:51.966346    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"]
Oct 29 22:40:51.972499 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:51.972471    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4d3168\"/\"test-model-4d3168-kserve-self-signed-certs\""
Oct 29 22:40:51.972499 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:51.972491    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4d3168\"/\"default-dockercfg-lm4hb\""
Oct 29 22:40:51.972758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:51.972538    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4d3168\"/\"kube-root-ca.crt\""
Oct 29 22:40:51.972758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:51.972562    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4d3168\"/\"openshift-service-ca.crt\""
Oct 29 22:40:52.013511 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.013470    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-tmp-dir\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.013965 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.013582    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2587k\" (UniqueName: \"kubernetes.io/projected/984111f3-9faf-43d2-8554-7bec87910f0b-kube-api-access-2587k\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.013965 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.013671    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-model-cache\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.013965 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.013850    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-dshm\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.014090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.014074    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-home\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.014147 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.014105    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/984111f3-9faf-43d2-8554-7bec87910f0b-tls-certs\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.115375 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.115335    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-tmp-dir\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.115550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.115463    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2587k\" (UniqueName: \"kubernetes.io/projected/984111f3-9faf-43d2-8554-7bec87910f0b-kube-api-access-2587k\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.115550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.115503    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-model-cache\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.115706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.115602    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-dshm\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.115777 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.115760    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-home\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.115902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.115793    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/984111f3-9faf-43d2-8554-7bec87910f0b-tls-certs\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.115902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.115792    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-tmp-dir\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.116035 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.115978    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-model-cache\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.116141 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.116117    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-home\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.118185 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.118153    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-dshm\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.118466 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.118435    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/984111f3-9faf-43d2-8554-7bec87910f0b-tls-certs\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.136679 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.136647    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2587k\" (UniqueName: \"kubernetes.io/projected/984111f3-9faf-43d2-8554-7bec87910f0b-kube-api-access-2587k\") pod \"test-model-4d3168-kserve-6778b8584c-2t9nc\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") " pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.277074 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.273664    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:52.665139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.644371    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"]
Oct 29 22:40:52.690154 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:52.690108    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8" podUID="cc440498-ac20-43de-b45a-3d0bcb2c312f" containerName="istio-proxy" containerID="cri-o://6da36e66fcb64f5c4d6845f8ddf2301f7c2d1110c6a44f7b75a62218090d4011" gracePeriod=30
Oct 29 22:40:53.009162 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:40:53.009130    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod984111f3_9faf_43d2_8554_7bec87910f0b.slice/crio-76879691a2a802010e22578bbd57f3a6e7a8e22efff16c1d2884dba27ec05c98 WatchSource:0}: Error finding container 76879691a2a802010e22578bbd57f3a6e7a8e22efff16c1d2884dba27ec05c98: Status 404 returned error can't find the container with id 76879691a2a802010e22578bbd57f3a6e7a8e22efff16c1d2884dba27ec05c98
Oct 29 22:40:53.009348 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:53.009325    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"]
Oct 29 22:40:53.999939 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:53.999902    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc" event={"ID":"984111f3-9faf-43d2-8554-7bec87910f0b","Type":"ContainerStarted","Data":"e299880ae898ec3a794ba36f619c271dfda0c409e1e136874f336b9477bc8dd6"}
Oct 29 22:40:54.000395 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:53.999948    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc" event={"ID":"984111f3-9faf-43d2-8554-7bec87910f0b","Type":"ContainerStarted","Data":"76879691a2a802010e22578bbd57f3a6e7a8e22efff16c1d2884dba27ec05c98"}
Oct 29 22:40:54.062436 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:54.062377    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc" podStartSLOduration=3.062361476 podStartE2EDuration="3.062361476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:54.059732584 +0000 UTC m=+1592.971176164" watchObservedRunningTime="2026-10-29 22:40:54.062361476 +0000 UTC m=+1592.973805080"
Oct 29 22:40:54.275894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:54.275182    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:40:54.927612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:54.927578    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:40:58.113408 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.113377    4559 generic.go:356] "Generic (PLEG): container finished" podID="cc440498-ac20-43de-b45a-3d0bcb2c312f" containerID="6da36e66fcb64f5c4d6845f8ddf2301f7c2d1110c6a44f7b75a62218090d4011" exitCode=0
Oct 29 22:40:58.113791 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.113533    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8" event={"ID":"cc440498-ac20-43de-b45a-3d0bcb2c312f","Type":"ContainerDied","Data":"6da36e66fcb64f5c4d6845f8ddf2301f7c2d1110c6a44f7b75a62218090d4011"}
Oct 29 22:40:58.174421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.174391    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:40:58.198876 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.198843    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc440498-ac20-43de-b45a-3d0bcb2c312f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc440498-ac20-43de-b45a-3d0bcb2c312f-istiod-ca-cert\") pod \"cc440498-ac20-43de-b45a-3d0bcb2c312f\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") "
Oct 29 22:40:58.199081 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.198917    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-podinfo\") pod \"cc440498-ac20-43de-b45a-3d0bcb2c312f\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") "
Oct 29 22:40:58.199081 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.198940    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-workload-socket\") pod \"cc440498-ac20-43de-b45a-3d0bcb2c312f\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") "
Oct 29 22:40:58.199081 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.198988    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-data\") pod \"cc440498-ac20-43de-b45a-3d0bcb2c312f\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") "
Oct 29 22:40:58.199247 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.199097    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc440498-ac20-43de-b45a-3d0bcb2c312f-kube-api-access-ldxwz\" (UniqueName: \"kubernetes.io/projected/cc440498-ac20-43de-b45a-3d0bcb2c312f-kube-api-access-ldxwz\") pod \"cc440498-ac20-43de-b45a-3d0bcb2c312f\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") "
Oct 29 22:40:58.199247 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.199174    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-token\" (UniqueName: \"kubernetes.io/projected/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-token\") pod \"cc440498-ac20-43de-b45a-3d0bcb2c312f\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") "
Oct 29 22:40:58.199247 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.199201    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-envoy\") pod \"cc440498-ac20-43de-b45a-3d0bcb2c312f\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") "
Oct 29 22:40:58.199381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.199241    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-workload-socket" pod "cc440498-ac20-43de-b45a-3d0bcb2c312f" (UID: "cc440498-ac20-43de-b45a-3d0bcb2c312f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:58.199381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.199289    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc440498-ac20-43de-b45a-3d0bcb2c312f-istiod-ca-cert" pod "cc440498-ac20-43de-b45a-3d0bcb2c312f" (UID: "cc440498-ac20-43de-b45a-3d0bcb2c312f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:58.199381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.199309    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-workload-certs\") pod \"cc440498-ac20-43de-b45a-3d0bcb2c312f\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") "
Oct 29 22:40:58.199552 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.199411    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-credential-socket\") pod \"cc440498-ac20-43de-b45a-3d0bcb2c312f\" (UID: \"cc440498-ac20-43de-b45a-3d0bcb2c312f\") "
Oct 29 22:40:58.199703 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.199677    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-data" pod "cc440498-ac20-43de-b45a-3d0bcb2c312f" (UID: "cc440498-ac20-43de-b45a-3d0bcb2c312f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:58.199903 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.199886    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-workload-certs" pod "cc440498-ac20-43de-b45a-3d0bcb2c312f" (UID: "cc440498-ac20-43de-b45a-3d0bcb2c312f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:58.200226 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.200154    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-credential-socket" pod "cc440498-ac20-43de-b45a-3d0bcb2c312f" (UID: "cc440498-ac20-43de-b45a-3d0bcb2c312f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:58.200463 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.200261    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:58.200463 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.200277    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc440498-ac20-43de-b45a-3d0bcb2c312f-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:58.200463 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.200285    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:58.200463 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.200293    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:58.202074 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.202030    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc440498-ac20-43de-b45a-3d0bcb2c312f-kube-api-access-ldxwz" pod "cc440498-ac20-43de-b45a-3d0bcb2c312f" (UID: "cc440498-ac20-43de-b45a-3d0bcb2c312f"). InnerVolumeSpecName "kube-api-access-ldxwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:58.202555 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.202532    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-podinfo" pod "cc440498-ac20-43de-b45a-3d0bcb2c312f" (UID: "cc440498-ac20-43de-b45a-3d0bcb2c312f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:40:58.203948 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.203917    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-token" pod "cc440498-ac20-43de-b45a-3d0bcb2c312f" (UID: "cc440498-ac20-43de-b45a-3d0bcb2c312f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:58.212167 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.212136    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-envoy" pod "cc440498-ac20-43de-b45a-3d0bcb2c312f" (UID: "cc440498-ac20-43de-b45a-3d0bcb2c312f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:58.302003 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.301973    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:58.302003 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.302005    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:58.302223 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.302022    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc440498-ac20-43de-b45a-3d0bcb2c312f-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:58.302223 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.302038    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc440498-ac20-43de-b45a-3d0bcb2c312f-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:58.302223 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:58.302051    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldxwz\" (UniqueName: \"kubernetes.io/projected/cc440498-ac20-43de-b45a-3d0bcb2c312f-kube-api-access-ldxwz\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:40:59.123155 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:59.123112    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8" event={"ID":"cc440498-ac20-43de-b45a-3d0bcb2c312f","Type":"ContainerDied","Data":"7ae0217528fc741a5bec7f89d9f8031f6759889a6f4af3f9a2603a66262f59d5"}
Oct 29 22:40:59.123155 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:59.123141    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"
Oct 29 22:40:59.123902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:59.123179    4559 scope.go:122] "RemoveContainer" containerID="6da36e66fcb64f5c4d6845f8ddf2301f7c2d1110c6a44f7b75a62218090d4011"
Oct 29 22:40:59.216842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:59.210017    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"]
Oct 29 22:40:59.216842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:59.212532    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-dnzl8"]
Oct 29 22:40:59.263833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:40:59.254502    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc440498-ac20-43de-b45a-3d0bcb2c312f" path="/var/lib/kubelet/pods/cc440498-ac20-43de-b45a-3d0bcb2c312f/volumes"
Oct 29 22:41:02.281838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:02.277903    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:41:02.773871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:02.773771    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:41:03.231732 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:03.231697    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:41:07.443981 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:07.443901    4559 patch_prober.go:29] interesting pod/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:41:07.444742 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:07.444700    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4" podUID="65fe44fa-7821-4581-9607-05d919e94273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:41:09.358122 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:09.358083    4559 generic.go:356] "Generic (PLEG): container finished" podID="65fe44fa-7821-4581-9607-05d919e94273" containerID="8524e8f9021ce677259bd2a7be6982af38b91d4322f62867409f352f8e3de2a8" exitCode=0
Oct 29 22:41:09.359026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:09.358996    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4" event={"ID":"65fe44fa-7821-4581-9607-05d919e94273","Type":"ContainerDied","Data":"8524e8f9021ce677259bd2a7be6982af38b91d4322f62867409f352f8e3de2a8"}
Oct 29 22:41:09.359821 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:09.359770    4559 scope.go:122] "RemoveContainer" containerID="8524e8f9021ce677259bd2a7be6982af38b91d4322f62867409f352f8e3de2a8"
Oct 29 22:41:09.366124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:09.366063    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:41:09.366436 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:09.366394    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:41:09.366553 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:09.366527    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:41:10.371186 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:10.371149    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4" event={"ID":"65fe44fa-7821-4581-9607-05d919e94273","Type":"ContainerStarted","Data":"ee6c01931c61f72ec9f143422e52652960e451c1a0ac352a7a7665fca26f4e0a"}
Oct 29 22:41:11.046839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:11.039608    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:41:11.048605 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:11.048569    4559 patch_prober.go:29] interesting pod/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 29 22:41:11.048891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:11.048863    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4" podUID="65fe44fa-7821-4581-9607-05d919e94273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 29 22:41:12.052982 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:12.052944    4559 patch_prober.go:29] interesting pod/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 29 22:41:12.053581 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:12.053548    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4" podUID="65fe44fa-7821-4581-9607-05d919e94273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 29 22:41:12.376907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:12.376133    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"]
Oct 29 22:41:12.376907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:12.376382    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6" podUID="ccce4424-021a-44ce-aa08-c1d46bf935a5" containerName="maas-api" containerID="cri-o://9089218d78dc5dcdbdd09e1d2710a866c0386cbc679133899e2cf5e8c80144a4" gracePeriod=30
Oct 29 22:41:13.049274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.041926    4559 patch_prober.go:29] interesting pod/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 29 22:41:13.049274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.041991    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4" podUID="65fe44fa-7821-4581-9607-05d919e94273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 29 22:41:13.654840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.650020    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:41:13.657945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.657463    4559 generic.go:356] "Generic (PLEG): container finished" podID="ccce4424-021a-44ce-aa08-c1d46bf935a5" containerID="9089218d78dc5dcdbdd09e1d2710a866c0386cbc679133899e2cf5e8c80144a4" exitCode=0
Oct 29 22:41:13.658315 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.658271    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6" event={"ID":"ccce4424-021a-44ce-aa08-c1d46bf935a5","Type":"ContainerDied","Data":"9089218d78dc5dcdbdd09e1d2710a866c0386cbc679133899e2cf5e8c80144a4"}
Oct 29 22:41:13.658393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.658333    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6" event={"ID":"ccce4424-021a-44ce-aa08-c1d46bf935a5","Type":"ContainerDied","Data":"bc6918afb7a86ff498a1550f2aa3a7f0d40c51b884a21cec04d361f410ae8e37"}
Oct 29 22:41:13.658393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.658356    4559 scope.go:122] "RemoveContainer" containerID="9089218d78dc5dcdbdd09e1d2710a866c0386cbc679133899e2cf5e8c80144a4"
Oct 29 22:41:13.716138 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.710723    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccce4424-021a-44ce-aa08-c1d46bf935a5-kube-api-access-jt29s\" (UniqueName: \"kubernetes.io/projected/ccce4424-021a-44ce-aa08-c1d46bf935a5-kube-api-access-jt29s\") pod \"ccce4424-021a-44ce-aa08-c1d46bf935a5\" (UID: \"ccce4424-021a-44ce-aa08-c1d46bf935a5\") "
Oct 29 22:41:13.716138 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.710984    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccce4424-021a-44ce-aa08-c1d46bf935a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccce4424-021a-44ce-aa08-c1d46bf935a5-openshift-service-ca\") pod \"ccce4424-021a-44ce-aa08-c1d46bf935a5\" (UID: \"ccce4424-021a-44ce-aa08-c1d46bf935a5\") "
Oct 29 22:41:13.716138 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.711055    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccce4424-021a-44ce-aa08-c1d46bf935a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccce4424-021a-44ce-aa08-c1d46bf935a5-metrics-certs\") pod \"ccce4424-021a-44ce-aa08-c1d46bf935a5\" (UID: \"ccce4424-021a-44ce-aa08-c1d46bf935a5\") "
Oct 29 22:41:13.716138 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.711085    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccce4424-021a-44ce-aa08-c1d46bf935a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccce4424-021a-44ce-aa08-c1d46bf935a5-maas-api-tls\") pod \"ccce4424-021a-44ce-aa08-c1d46bf935a5\" (UID: \"ccce4424-021a-44ce-aa08-c1d46bf935a5\") "
Oct 29 22:41:13.716138 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.713281    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccce4424-021a-44ce-aa08-c1d46bf935a5-openshift-service-ca" pod "ccce4424-021a-44ce-aa08-c1d46bf935a5" (UID: "ccce4424-021a-44ce-aa08-c1d46bf935a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:41:13.721983 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.721942    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccce4424-021a-44ce-aa08-c1d46bf935a5-metrics-certs" pod "ccce4424-021a-44ce-aa08-c1d46bf935a5" (UID: "ccce4424-021a-44ce-aa08-c1d46bf935a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:41:13.723110 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.722917    4559 scope.go:122] "RemoveContainer" containerID="9089218d78dc5dcdbdd09e1d2710a866c0386cbc679133899e2cf5e8c80144a4"
Oct 29 22:41:13.723110 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.722970    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccce4424-021a-44ce-aa08-c1d46bf935a5-kube-api-access-jt29s" pod "ccce4424-021a-44ce-aa08-c1d46bf935a5" (UID: "ccce4424-021a-44ce-aa08-c1d46bf935a5"). InnerVolumeSpecName "kube-api-access-jt29s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:13.723110 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.723007    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccce4424-021a-44ce-aa08-c1d46bf935a5-maas-api-tls" pod "ccce4424-021a-44ce-aa08-c1d46bf935a5" (UID: "ccce4424-021a-44ce-aa08-c1d46bf935a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:41:13.723372 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:41:13.723343    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9089218d78dc5dcdbdd09e1d2710a866c0386cbc679133899e2cf5e8c80144a4\": container with ID starting with 9089218d78dc5dcdbdd09e1d2710a866c0386cbc679133899e2cf5e8c80144a4 not found: ID does not exist" containerID="9089218d78dc5dcdbdd09e1d2710a866c0386cbc679133899e2cf5e8c80144a4"
Oct 29 22:41:13.723432 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.723387    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9089218d78dc5dcdbdd09e1d2710a866c0386cbc679133899e2cf5e8c80144a4"} err="failed to get container status \"9089218d78dc5dcdbdd09e1d2710a866c0386cbc679133899e2cf5e8c80144a4\": rpc error: code = NotFound desc = could not find container \"9089218d78dc5dcdbdd09e1d2710a866c0386cbc679133899e2cf5e8c80144a4\": container with ID starting with 9089218d78dc5dcdbdd09e1d2710a866c0386cbc679133899e2cf5e8c80144a4 not found: ID does not exist"
Oct 29 22:41:13.769727 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:41:13.769690    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:41:13.820181 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.820149    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt29s\" (UniqueName: \"kubernetes.io/projected/ccce4424-021a-44ce-aa08-c1d46bf935a5-kube-api-access-jt29s\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:13.820181 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.820180    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccce4424-021a-44ce-aa08-c1d46bf935a5-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:13.820440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.820193    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccce4424-021a-44ce-aa08-c1d46bf935a5-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:13.820440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:13.820201    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccce4424-021a-44ce-aa08-c1d46bf935a5-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:14.060892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:14.059829    4559 patch_prober.go:29] interesting pod/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 29 22:41:14.060892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:14.059890    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4" podUID="65fe44fa-7821-4581-9607-05d919e94273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 29 22:41:14.676341 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:14.676303    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"
Oct 29 22:41:14.767740 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:14.767460    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"]
Oct 29 22:41:14.773028 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:14.772996    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39322f-595467c98-dvbw6"]
Oct 29 22:41:14.890519 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:14.890468    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"]
Oct 29 22:41:14.892263 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:14.892230    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6" podUID="2e5ef4a7-965a-4695-834d-37c4e8cf0beb" containerName="maas-api" containerID="cri-o://fdb0d29e8f9b44d0e5dba075599f17f6761da55457d8bd20435786cc01f1b4c2" gracePeriod=30
Oct 29 22:41:15.067836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.067016    4559 patch_prober.go:29] interesting pod/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 29 22:41:15.067836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.067080    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4" podUID="65fe44fa-7821-4581-9607-05d919e94273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 29 22:41:15.455918 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.448353    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccce4424-021a-44ce-aa08-c1d46bf935a5" path="/var/lib/kubelet/pods/ccce4424-021a-44ce-aa08-c1d46bf935a5/volumes"
Oct 29 22:41:15.768892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.759078    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:41:15.768892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.761948    4559 generic.go:356] "Generic (PLEG): container finished" podID="2e5ef4a7-965a-4695-834d-37c4e8cf0beb" containerID="fdb0d29e8f9b44d0e5dba075599f17f6761da55457d8bd20435786cc01f1b4c2" exitCode=0
Oct 29 22:41:15.768892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.762043    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6" event={"ID":"2e5ef4a7-965a-4695-834d-37c4e8cf0beb","Type":"ContainerDied","Data":"fdb0d29e8f9b44d0e5dba075599f17f6761da55457d8bd20435786cc01f1b4c2"}
Oct 29 22:41:15.768892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.762114    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6" event={"ID":"2e5ef4a7-965a-4695-834d-37c4e8cf0beb","Type":"ContainerDied","Data":"5a846dd590ad4ce98a912721aac51104ffe0a926fab95658e91b6cd842e15332"}
Oct 29 22:41:15.768892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.762160    4559 scope.go:122] "RemoveContainer" containerID="fdb0d29e8f9b44d0e5dba075599f17f6761da55457d8bd20435786cc01f1b4c2"
Oct 29 22:41:15.808603 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.808536    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-openshift-service-ca\") pod \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\" (UID: \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\") "
Oct 29 22:41:15.809506 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.809481    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-maas-api-tls\") pod \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\" (UID: \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\") "
Oct 29 22:41:15.810013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.809992    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-openshift-service-ca" pod "2e5ef4a7-965a-4695-834d-37c4e8cf0beb" (UID: "2e5ef4a7-965a-4695-834d-37c4e8cf0beb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:41:15.810245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.810229    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-metrics-certs\") pod \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\" (UID: \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\") "
Oct 29 22:41:15.831899 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.831860    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-kube-api-access-t4q7t\" (UniqueName: \"kubernetes.io/projected/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-kube-api-access-t4q7t\") pod \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\" (UID: \"2e5ef4a7-965a-4695-834d-37c4e8cf0beb\") "
Oct 29 22:41:15.850707 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.850673    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:15.867305 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.867265    4559 scope.go:122] "RemoveContainer" containerID="fdb0d29e8f9b44d0e5dba075599f17f6761da55457d8bd20435786cc01f1b4c2"
Oct 29 22:41:15.877159 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:41:15.877113    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb0d29e8f9b44d0e5dba075599f17f6761da55457d8bd20435786cc01f1b4c2\": container with ID starting with fdb0d29e8f9b44d0e5dba075599f17f6761da55457d8bd20435786cc01f1b4c2 not found: ID does not exist" containerID="fdb0d29e8f9b44d0e5dba075599f17f6761da55457d8bd20435786cc01f1b4c2"
Oct 29 22:41:15.877450 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.877409    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb0d29e8f9b44d0e5dba075599f17f6761da55457d8bd20435786cc01f1b4c2"} err="failed to get container status \"fdb0d29e8f9b44d0e5dba075599f17f6761da55457d8bd20435786cc01f1b4c2\": rpc error: code = NotFound desc = could not find container \"fdb0d29e8f9b44d0e5dba075599f17f6761da55457d8bd20435786cc01f1b4c2\": container with ID starting with fdb0d29e8f9b44d0e5dba075599f17f6761da55457d8bd20435786cc01f1b4c2 not found: ID does not exist"
Oct 29 22:41:15.901057 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.893392    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-kube-api-access-t4q7t" pod "2e5ef4a7-965a-4695-834d-37c4e8cf0beb" (UID: "2e5ef4a7-965a-4695-834d-37c4e8cf0beb"). InnerVolumeSpecName "kube-api-access-t4q7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:15.978089 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.962016    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-maas-api-tls" pod "2e5ef4a7-965a-4695-834d-37c4e8cf0beb" (UID: "2e5ef4a7-965a-4695-834d-37c4e8cf0beb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:41:15.981844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.979560    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:15.981844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:15.979600    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4q7t\" (UniqueName: \"kubernetes.io/projected/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-kube-api-access-t4q7t\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:16.107942 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:16.101869    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-metrics-certs" pod "2e5ef4a7-965a-4695-834d-37c4e8cf0beb" (UID: "2e5ef4a7-965a-4695-834d-37c4e8cf0beb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:41:16.107942 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:16.105930    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5ef4a7-965a-4695-834d-37c4e8cf0beb-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:16.154891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:16.125877    4559 patch_prober.go:29] interesting pod/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 29 22:41:16.154891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:16.125945    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4" podUID="65fe44fa-7821-4581-9607-05d919e94273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 29 22:41:16.239947 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:16.239898    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc" podUID="984111f3-9faf-43d2-8554-7bec87910f0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:41:16.257996 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:16.257951    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:41:16.773219 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:16.773188    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"
Oct 29 22:41:16.830365 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:16.830323    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"]
Oct 29 22:41:17.052285 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:17.052194    4559 patch_prober.go:29] interesting pod/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 29 22:41:17.066937 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:17.066890    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4" podUID="65fe44fa-7821-4581-9607-05d919e94273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 29 22:41:17.178875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:17.176940    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d1777-774f978df4-74jq6"]
Oct 29 22:41:17.402139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:17.398700    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e5ef4a7-965a-4695-834d-37c4e8cf0beb" path="/var/lib/kubelet/pods/2e5ef4a7-965a-4695-834d-37c4e8cf0beb/volumes"
Oct 29 22:41:18.149867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:18.147305    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:41:18.149867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:18.148198    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:41:18.151870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:18.150918    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:41:21.928919 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:21.928881    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"]
Oct 29 22:41:21.948841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:21.931794    4559 cpu_manager.go:414] "containerMap: removing container" podUID="cc440498-ac20-43de-b45a-3d0bcb2c312f" containerName="istio-proxy"
Oct 29 22:41:21.953938 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:21.953903    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc440498-ac20-43de-b45a-3d0bcb2c312f" containerName="istio-proxy"
Oct 29 22:41:21.954227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:21.954209    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ccce4424-021a-44ce-aa08-c1d46bf935a5" containerName="maas-api"
Oct 29 22:41:21.954338 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:21.954326    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccce4424-021a-44ce-aa08-c1d46bf935a5" containerName="maas-api"
Oct 29 22:41:21.954474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:21.954460    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2e5ef4a7-965a-4695-834d-37c4e8cf0beb" containerName="maas-api"
Oct 29 22:41:21.954563 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:21.954552    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5ef4a7-965a-4695-834d-37c4e8cf0beb" containerName="maas-api"
Oct 29 22:41:21.954944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:21.954924    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccce4424-021a-44ce-aa08-c1d46bf935a5" containerName="maas-api"
Oct 29 22:41:21.955079 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:21.955067    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e5ef4a7-965a-4695-834d-37c4e8cf0beb" containerName="maas-api"
Oct 29 22:41:21.955177 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:21.955167    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc440498-ac20-43de-b45a-3d0bcb2c312f" containerName="istio-proxy"
Oct 29 22:41:22.170477 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.170442    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"]
Oct 29 22:41:22.170871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.170787    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.187953 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.187926    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c296a3\"/\"default-dockercfg-qmzlh\""
Oct 29 22:41:22.190924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.189183    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c296a3\"/\"openshift-service-ca.crt\""
Oct 29 22:41:22.199132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.199107    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c296a3\"/\"kube-root-ca.crt\""
Oct 29 22:41:22.199560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.199544    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c296a3\"/\"test-model-c296a3-kserve-self-signed-certs\""
Oct 29 22:41:22.271838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.251262    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjcdv\" (UniqueName: \"kubernetes.io/projected/38887980-8bf2-4efd-8f38-7a7fda65dadf-kube-api-access-xjcdv\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.271838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.251326    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-dshm\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.271838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.251497    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-model-cache\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.271838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.251686    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-home\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.271838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.251857    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-tmp-dir\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.271838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.251902    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38887980-8bf2-4efd-8f38-7a7fda65dadf-tls-certs\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.379046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.365996    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-tmp-dir\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.379046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.366077    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38887980-8bf2-4efd-8f38-7a7fda65dadf-tls-certs\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.379046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.371266    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-tmp-dir\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.379046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.376365    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjcdv\" (UniqueName: \"kubernetes.io/projected/38887980-8bf2-4efd-8f38-7a7fda65dadf-kube-api-access-xjcdv\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.379046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.376840    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-dshm\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.379046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.377449    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-model-cache\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.379046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.377776    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-model-cache\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.379046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.378081    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-home\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.379046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.378545    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-home\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.426056 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.405655    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-dshm\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.429880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.428189    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjcdv\" (UniqueName: \"kubernetes.io/projected/38887980-8bf2-4efd-8f38-7a7fda65dadf-kube-api-access-xjcdv\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.477436 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.471198    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38887980-8bf2-4efd-8f38-7a7fda65dadf-tls-certs\") pod \"test-model-c296a3-kserve-56d8758455-v2cnk\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") " pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:22.634163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:22.631412    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:23.012047 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:41:23.012016    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38887980_8bf2_4efd_8f38_7a7fda65dadf.slice/crio-2f6929fce6edf002ca7f5442521abbebe3606b0398ab5de336ee1974170dd2c6 WatchSource:0}: Error finding container 2f6929fce6edf002ca7f5442521abbebe3606b0398ab5de336ee1974170dd2c6: Status 404 returned error can't find the container with id 2f6929fce6edf002ca7f5442521abbebe3606b0398ab5de336ee1974170dd2c6
Oct 29 22:41:23.013385 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:23.013351    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"]
Oct 29 22:41:23.857428 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:23.857388    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk" event={"ID":"38887980-8bf2-4efd-8f38-7a7fda65dadf","Type":"ContainerStarted","Data":"7e1fad7bbbcb374b8b799830111bb2b034944e654dba1a00d07205bdaae10492"}
Oct 29 22:41:23.857428 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:23.857439    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk" event={"ID":"38887980-8bf2-4efd-8f38-7a7fda65dadf","Type":"ContainerStarted","Data":"2f6929fce6edf002ca7f5442521abbebe3606b0398ab5de336ee1974170dd2c6"}
Oct 29 22:41:23.901894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:23.901829    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk" podStartSLOduration=2.901791986 podStartE2EDuration="2.901791986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:23.897527285 +0000 UTC m=+1622.808970834" watchObservedRunningTime="2026-10-29 22:41:23.901791986 +0000 UTC m=+1622.813235534"
Oct 29 22:41:24.631859 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:24.631824    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:25.182461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:25.181087    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"]
Oct 29 22:41:25.182461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:25.181374    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv" podUID="5d1bb1ea-189a-4033-b702-7fba8db86cdf" containerName="istio-proxy" containerID="cri-o://f0ebdfc5b88f77117f530d3be3ced813f1cebaf535b2b20f5031be77d46bc2b3" gracePeriod=30
Oct 29 22:41:25.754440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:25.753498    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"]
Oct 29 22:41:25.754440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:25.753758    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh" podUID="4042e16a-2f51-4588-b54c-bfce92dd57cc" containerName="istio-proxy" containerID="cri-o://55a48a5177e24c35a5eb9f74e02da0460cab9012197a0fb1ea4c63042803d8bf" gracePeriod=30
Oct 29 22:41:25.765530 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:25.765465    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"]
Oct 29 22:41:25.766528 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:25.765756    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8" podUID="18d0c0ae-111d-4f92-ac13-58201176aa4c" containerName="istio-proxy" containerID="cri-o://e4e583091db7b0c2965c0f888d4d74a512706f89f013bb25df98416083cb5a85" gracePeriod=30
Oct 29 22:41:25.772569 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:25.772535    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"]
Oct 29 22:41:25.772898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:25.772870    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4" podUID="724aa157-d061-4f4d-8bcd-72d474e7207d" containerName="istio-proxy" containerID="cri-o://65879f6d2226944f10890f9ed3492b76361708fcb579eb30dec0e317e97d5d90" gracePeriod=30
Oct 29 22:41:28.041542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.041505    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"]
Oct 29 22:41:28.050952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.047358    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.065380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.065345    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"]
Oct 29 22:41:28.070200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.070138    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c1750b\"/\"kube-root-ca.crt\""
Oct 29 22:41:28.070498 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.070446    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c1750b\"/\"openshift-service-ca.crt\""
Oct 29 22:41:28.073003 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.071128    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c1750b\"/\"test-model-c1750b-kserve-self-signed-certs\""
Oct 29 22:41:28.080058 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.080018    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c1750b\"/\"default-dockercfg-jxn44\""
Oct 29 22:41:28.184944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.184907    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-model-cache\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.184944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.184948    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s7p4\" (UniqueName: \"kubernetes.io/projected/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-kube-api-access-9s7p4\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.185186 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.185109    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-dshm\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.185186 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.185159    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-home\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.185288 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.185272    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-tls-certs\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.185358 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.185343    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-tmp-dir\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.286870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.286834    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-tls-certs\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.287080 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.286929    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-tmp-dir\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.287080 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.286988    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-model-cache\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.287080 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.287014    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7p4\" (UniqueName: \"kubernetes.io/projected/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-kube-api-access-9s7p4\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.287265 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.287128    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-dshm\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.287265 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.287199    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-home\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.287468 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.287442    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-tmp-dir\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.287523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.287489    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-model-cache\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.287565 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.287497    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-home\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.289770 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.289742    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-dshm\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.289938 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.289747    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-tls-certs\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.307127 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.307055    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7p4\" (UniqueName: \"kubernetes.io/projected/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-kube-api-access-9s7p4\") pod \"test-model-c1750b-kserve-7969d8bf9f-dj2m4\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") " pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.374062 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.373686    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:28.928369 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:41:28.928336    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode44215a6_17c6_48e9_b4d6_123b1ef1ac12.slice/crio-6596d5122946fb15231892a69206708cd05d98359edf1adace52d3c5988a54b0 WatchSource:0}: Error finding container 6596d5122946fb15231892a69206708cd05d98359edf1adace52d3c5988a54b0: Status 404 returned error can't find the container with id 6596d5122946fb15231892a69206708cd05d98359edf1adace52d3c5988a54b0
Oct 29 22:41:28.928715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:28.928682    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"]
Oct 29 22:41:29.029744 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:29.029709    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4" event={"ID":"e44215a6-17c6-48e9-b4d6-123b1ef1ac12","Type":"ContainerStarted","Data":"e3b338131e0b9db5be5ae269715b167dd318f713e8b24d3c669fadee0e564705"}
Oct 29 22:41:29.029933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:29.029754    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4" event={"ID":"e44215a6-17c6-48e9-b4d6-123b1ef1ac12","Type":"ContainerStarted","Data":"6596d5122946fb15231892a69206708cd05d98359edf1adace52d3c5988a54b0"}
Oct 29 22:41:29.073180 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:29.073120    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4" podStartSLOduration=1.073103409 podStartE2EDuration="1.073103409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:29.070590141 +0000 UTC m=+1627.982033715" watchObservedRunningTime="2026-10-29 22:41:29.073103409 +0000 UTC m=+1627.984546964"
Oct 29 22:41:29.374677 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:29.374587    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:30.500541 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.500518    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:41:30.578772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.577137    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1bb1ea-189a-4033-b702-7fba8db86cdf-kube-api-access-kcx6t\" (UniqueName: \"kubernetes.io/projected/5d1bb1ea-189a-4033-b702-7fba8db86cdf-kube-api-access-kcx6t\") pod \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") "
Oct 29 22:41:30.578772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.577220    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-token\" (UniqueName: \"kubernetes.io/projected/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-token\") pod \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") "
Oct 29 22:41:30.578772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.577394    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-data\") pod \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") "
Oct 29 22:41:30.578772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.577433    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-workload-certs\") pod \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") "
Oct 29 22:41:30.578772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.577474    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-workload-socket\") pod \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") "
Oct 29 22:41:30.578772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.577562    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istiod-ca-cert\") pod \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") "
Oct 29 22:41:30.578772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.577646    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-podinfo\") pod \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") "
Oct 29 22:41:30.578772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.577746    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-envoy\") pod \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") "
Oct 29 22:41:30.578772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.577894    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-credential-socket\") pod \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\" (UID: \"5d1bb1ea-189a-4033-b702-7fba8db86cdf\") "
Oct 29 22:41:30.579396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.579162    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-workload-certs" pod "5d1bb1ea-189a-4033-b702-7fba8db86cdf" (UID: "5d1bb1ea-189a-4033-b702-7fba8db86cdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:30.579928 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.579828    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istiod-ca-cert" pod "5d1bb1ea-189a-4033-b702-7fba8db86cdf" (UID: "5d1bb1ea-189a-4033-b702-7fba8db86cdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:41:30.580196 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.580070    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-workload-socket" pod "5d1bb1ea-189a-4033-b702-7fba8db86cdf" (UID: "5d1bb1ea-189a-4033-b702-7fba8db86cdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:30.580722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.580469    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-data" pod "5d1bb1ea-189a-4033-b702-7fba8db86cdf" (UID: "5d1bb1ea-189a-4033-b702-7fba8db86cdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:30.581087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.581066    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-credential-socket" pod "5d1bb1ea-189a-4033-b702-7fba8db86cdf" (UID: "5d1bb1ea-189a-4033-b702-7fba8db86cdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:30.583664 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.583629    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-token" pod "5d1bb1ea-189a-4033-b702-7fba8db86cdf" (UID: "5d1bb1ea-189a-4033-b702-7fba8db86cdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:30.584673 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.584646    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1bb1ea-189a-4033-b702-7fba8db86cdf-kube-api-access-kcx6t" pod "5d1bb1ea-189a-4033-b702-7fba8db86cdf" (UID: "5d1bb1ea-189a-4033-b702-7fba8db86cdf"). InnerVolumeSpecName "kube-api-access-kcx6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:30.584921 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.584886    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-envoy" pod "5d1bb1ea-189a-4033-b702-7fba8db86cdf" (UID: "5d1bb1ea-189a-4033-b702-7fba8db86cdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:30.585327 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.585304    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-podinfo" pod "5d1bb1ea-189a-4033-b702-7fba8db86cdf" (UID: "5d1bb1ea-189a-4033-b702-7fba8db86cdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:41:30.681781 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.681739    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:30.681781 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.681782    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcx6t\" (UniqueName: \"kubernetes.io/projected/5d1bb1ea-189a-4033-b702-7fba8db86cdf-kube-api-access-kcx6t\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:30.682026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.681798    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:30.682026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.681830    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:30.682026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.681843    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:30.682026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.681856    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:30.682026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.681868    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:30.682026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.681880    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:30.682026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:30.681892    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1bb1ea-189a-4033-b702-7fba8db86cdf-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.065764 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.065727    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:41:31.089196 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.089161    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4042e16a-2f51-4588-b54c-bfce92dd57cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4042e16a-2f51-4588-b54c-bfce92dd57cc-istiod-ca-cert\") pod \"4042e16a-2f51-4588-b54c-bfce92dd57cc\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") "
Oct 29 22:41:31.089370 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.089247    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-envoy\") pod \"4042e16a-2f51-4588-b54c-bfce92dd57cc\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") "
Oct 29 22:41:31.089370 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.089354    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-workload-certs\") pod \"4042e16a-2f51-4588-b54c-bfce92dd57cc\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") "
Oct 29 22:41:31.089487 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.089446    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-data\") pod \"4042e16a-2f51-4588-b54c-bfce92dd57cc\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") "
Oct 29 22:41:31.089487 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.089480    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-credential-socket\") pod \"4042e16a-2f51-4588-b54c-bfce92dd57cc\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") "
Oct 29 22:41:31.089605 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.089541    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-podinfo\") pod \"4042e16a-2f51-4588-b54c-bfce92dd57cc\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") "
Oct 29 22:41:31.089605 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.089597    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-workload-socket\") pod \"4042e16a-2f51-4588-b54c-bfce92dd57cc\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") "
Oct 29 22:41:31.089706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.089618    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4042e16a-2f51-4588-b54c-bfce92dd57cc-istiod-ca-cert" pod "4042e16a-2f51-4588-b54c-bfce92dd57cc" (UID: "4042e16a-2f51-4588-b54c-bfce92dd57cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:41:31.089706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.089678    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-token\" (UniqueName: \"kubernetes.io/projected/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-token\") pod \"4042e16a-2f51-4588-b54c-bfce92dd57cc\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") "
Oct 29 22:41:31.089831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.089722    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4042e16a-2f51-4588-b54c-bfce92dd57cc-kube-api-access-2h9dw\" (UniqueName: \"kubernetes.io/projected/4042e16a-2f51-4588-b54c-bfce92dd57cc-kube-api-access-2h9dw\") pod \"4042e16a-2f51-4588-b54c-bfce92dd57cc\" (UID: \"4042e16a-2f51-4588-b54c-bfce92dd57cc\") "
Oct 29 22:41:31.090119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.090096    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-credential-socket" pod "4042e16a-2f51-4588-b54c-bfce92dd57cc" (UID: "4042e16a-2f51-4588-b54c-bfce92dd57cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.090403 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.090385    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4042e16a-2f51-4588-b54c-bfce92dd57cc-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.090523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.090408    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.090641 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.090620    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-data" pod "4042e16a-2f51-4588-b54c-bfce92dd57cc" (UID: "4042e16a-2f51-4588-b54c-bfce92dd57cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.090727 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.090639    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-workload-certs" pod "4042e16a-2f51-4588-b54c-bfce92dd57cc" (UID: "4042e16a-2f51-4588-b54c-bfce92dd57cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.090882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.090846    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-workload-socket" pod "4042e16a-2f51-4588-b54c-bfce92dd57cc" (UID: "4042e16a-2f51-4588-b54c-bfce92dd57cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.091984 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.091955    4559 generic.go:356] "Generic (PLEG): container finished" podID="18d0c0ae-111d-4f92-ac13-58201176aa4c" containerID="e4e583091db7b0c2965c0f888d4d74a512706f89f013bb25df98416083cb5a85" exitCode=0
Oct 29 22:41:31.092212 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.092136    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8" event={"ID":"18d0c0ae-111d-4f92-ac13-58201176aa4c","Type":"ContainerDied","Data":"e4e583091db7b0c2965c0f888d4d74a512706f89f013bb25df98416083cb5a85"}
Oct 29 22:41:31.093796 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.093775    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-envoy" pod "4042e16a-2f51-4588-b54c-bfce92dd57cc" (UID: "4042e16a-2f51-4588-b54c-bfce92dd57cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.093921 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.093803    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-podinfo" pod "4042e16a-2f51-4588-b54c-bfce92dd57cc" (UID: "4042e16a-2f51-4588-b54c-bfce92dd57cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:41:31.094647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.094621    4559 generic.go:356] "Generic (PLEG): container finished" podID="724aa157-d061-4f4d-8bcd-72d474e7207d" containerID="65879f6d2226944f10890f9ed3492b76361708fcb579eb30dec0e317e97d5d90" exitCode=0
Oct 29 22:41:31.094753 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.094688    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4" event={"ID":"724aa157-d061-4f4d-8bcd-72d474e7207d","Type":"ContainerDied","Data":"65879f6d2226944f10890f9ed3492b76361708fcb579eb30dec0e317e97d5d90"}
Oct 29 22:41:31.103879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.096847    4559 generic.go:356] "Generic (PLEG): container finished" podID="5d1bb1ea-189a-4033-b702-7fba8db86cdf" containerID="f0ebdfc5b88f77117f530d3be3ced813f1cebaf535b2b20f5031be77d46bc2b3" exitCode=0
Oct 29 22:41:31.103879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.097006    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv" event={"ID":"5d1bb1ea-189a-4033-b702-7fba8db86cdf","Type":"ContainerDied","Data":"f0ebdfc5b88f77117f530d3be3ced813f1cebaf535b2b20f5031be77d46bc2b3"}
Oct 29 22:41:31.103879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.097039    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv" event={"ID":"5d1bb1ea-189a-4033-b702-7fba8db86cdf","Type":"ContainerDied","Data":"40013671cdf424ece95299c6a59da1f570ca49679562263f34940057f318b113"}
Oct 29 22:41:31.103879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.097065    4559 scope.go:122] "RemoveContainer" containerID="f0ebdfc5b88f77117f530d3be3ced813f1cebaf535b2b20f5031be77d46bc2b3"
Oct 29 22:41:31.103879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.097233    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"
Oct 29 22:41:31.103879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.098982    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-token" pod "4042e16a-2f51-4588-b54c-bfce92dd57cc" (UID: "4042e16a-2f51-4588-b54c-bfce92dd57cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:31.103879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.099331    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4042e16a-2f51-4588-b54c-bfce92dd57cc-kube-api-access-2h9dw" pod "4042e16a-2f51-4588-b54c-bfce92dd57cc" (UID: "4042e16a-2f51-4588-b54c-bfce92dd57cc"). InnerVolumeSpecName "kube-api-access-2h9dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:31.106125 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.105604    4559 generic.go:356] "Generic (PLEG): container finished" podID="4042e16a-2f51-4588-b54c-bfce92dd57cc" containerID="55a48a5177e24c35a5eb9f74e02da0460cab9012197a0fb1ea4c63042803d8bf" exitCode=0
Oct 29 22:41:31.106125 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.105967    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh" event={"ID":"4042e16a-2f51-4588-b54c-bfce92dd57cc","Type":"ContainerDied","Data":"55a48a5177e24c35a5eb9f74e02da0460cab9012197a0fb1ea4c63042803d8bf"}
Oct 29 22:41:31.106125 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.106024    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh" event={"ID":"4042e16a-2f51-4588-b54c-bfce92dd57cc","Type":"ContainerDied","Data":"547a56e34db6d78bf2376446b3829774cb752a4b5667580fe65f1357825bdd73"}
Oct 29 22:41:31.107136 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.106465    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"
Oct 29 22:41:31.149619 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.149594    4559 scope.go:122] "RemoveContainer" containerID="7ffc1e613e0ea35e105c2caa1119cf09feeebafd0223e8a9b3ed0582f498cb3d"
Oct 29 22:41:31.168228 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.168200    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:41:31.169030 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.169009    4559 scope.go:122] "RemoveContainer" containerID="f0ebdfc5b88f77117f530d3be3ced813f1cebaf535b2b20f5031be77d46bc2b3"
Oct 29 22:41:31.173030 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:41:31.172992    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ebdfc5b88f77117f530d3be3ced813f1cebaf535b2b20f5031be77d46bc2b3\": container with ID starting with f0ebdfc5b88f77117f530d3be3ced813f1cebaf535b2b20f5031be77d46bc2b3 not found: ID does not exist" containerID="f0ebdfc5b88f77117f530d3be3ced813f1cebaf535b2b20f5031be77d46bc2b3"
Oct 29 22:41:31.173212 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.173187    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ebdfc5b88f77117f530d3be3ced813f1cebaf535b2b20f5031be77d46bc2b3"} err="failed to get container status \"f0ebdfc5b88f77117f530d3be3ced813f1cebaf535b2b20f5031be77d46bc2b3\": rpc error: code = NotFound desc = could not find container \"f0ebdfc5b88f77117f530d3be3ced813f1cebaf535b2b20f5031be77d46bc2b3\": container with ID starting with f0ebdfc5b88f77117f530d3be3ced813f1cebaf535b2b20f5031be77d46bc2b3 not found: ID does not exist"
Oct 29 22:41:31.173305 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.173295    4559 scope.go:122] "RemoveContainer" containerID="7ffc1e613e0ea35e105c2caa1119cf09feeebafd0223e8a9b3ed0582f498cb3d"
Oct 29 22:41:31.173732 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:41:31.173706    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ffc1e613e0ea35e105c2caa1119cf09feeebafd0223e8a9b3ed0582f498cb3d\": container with ID starting with 7ffc1e613e0ea35e105c2caa1119cf09feeebafd0223e8a9b3ed0582f498cb3d not found: ID does not exist" containerID="7ffc1e613e0ea35e105c2caa1119cf09feeebafd0223e8a9b3ed0582f498cb3d"
Oct 29 22:41:31.174011 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.173987    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ffc1e613e0ea35e105c2caa1119cf09feeebafd0223e8a9b3ed0582f498cb3d"} err="failed to get container status \"7ffc1e613e0ea35e105c2caa1119cf09feeebafd0223e8a9b3ed0582f498cb3d\": rpc error: code = NotFound desc = could not find container \"7ffc1e613e0ea35e105c2caa1119cf09feeebafd0223e8a9b3ed0582f498cb3d\": container with ID starting with 7ffc1e613e0ea35e105c2caa1119cf09feeebafd0223e8a9b3ed0582f498cb3d not found: ID does not exist"
Oct 29 22:41:31.174122 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.174112    4559 scope.go:122] "RemoveContainer" containerID="55a48a5177e24c35a5eb9f74e02da0460cab9012197a0fb1ea4c63042803d8bf"
Oct 29 22:41:31.191152 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.191129    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d0c0ae-111d-4f92-ac13-58201176aa4c-kube-api-access-vpg2b\" (UniqueName: \"kubernetes.io/projected/18d0c0ae-111d-4f92-ac13-58201176aa4c-kube-api-access-vpg2b\") pod \"18d0c0ae-111d-4f92-ac13-58201176aa4c\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") "
Oct 29 22:41:31.191793 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.191778    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-data\") pod \"18d0c0ae-111d-4f92-ac13-58201176aa4c\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") "
Oct 29 22:41:31.192987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.191654    4559 scope.go:122] "RemoveContainer" containerID="55a48a5177e24c35a5eb9f74e02da0460cab9012197a0fb1ea4c63042803d8bf"
Oct 29 22:41:31.192987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.192336    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-data" pod "18d0c0ae-111d-4f92-ac13-58201176aa4c" (UID: "18d0c0ae-111d-4f92-ac13-58201176aa4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.192987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.192970    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-podinfo\") pod \"18d0c0ae-111d-4f92-ac13-58201176aa4c\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") "
Oct 29 22:41:31.193270 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.193093    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-credential-socket\") pod \"18d0c0ae-111d-4f92-ac13-58201176aa4c\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") "
Oct 29 22:41:31.193429 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.193345    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-workload-certs\") pod \"18d0c0ae-111d-4f92-ac13-58201176aa4c\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") "
Oct 29 22:41:31.193429 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.193374    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-credential-socket" pod "18d0c0ae-111d-4f92-ac13-58201176aa4c" (UID: "18d0c0ae-111d-4f92-ac13-58201176aa4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.193550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.193431    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-workload-socket\") pod \"18d0c0ae-111d-4f92-ac13-58201176aa4c\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") "
Oct 29 22:41:31.193550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.193465    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-token\" (UniqueName: \"kubernetes.io/projected/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-token\") pod \"18d0c0ae-111d-4f92-ac13-58201176aa4c\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") "
Oct 29 22:41:31.193550 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:41:31.193506    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a48a5177e24c35a5eb9f74e02da0460cab9012197a0fb1ea4c63042803d8bf\": container with ID starting with 55a48a5177e24c35a5eb9f74e02da0460cab9012197a0fb1ea4c63042803d8bf not found: ID does not exist" containerID="55a48a5177e24c35a5eb9f74e02da0460cab9012197a0fb1ea4c63042803d8bf"
Oct 29 22:41:31.193715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.193537    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a48a5177e24c35a5eb9f74e02da0460cab9012197a0fb1ea4c63042803d8bf"} err="failed to get container status \"55a48a5177e24c35a5eb9f74e02da0460cab9012197a0fb1ea4c63042803d8bf\": rpc error: code = NotFound desc = could not find container \"55a48a5177e24c35a5eb9f74e02da0460cab9012197a0fb1ea4c63042803d8bf\": container with ID starting with 55a48a5177e24c35a5eb9f74e02da0460cab9012197a0fb1ea4c63042803d8bf not found: ID does not exist"
Oct 29 22:41:31.193715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.193621    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18d0c0ae-111d-4f92-ac13-58201176aa4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d0c0ae-111d-4f92-ac13-58201176aa4c-istiod-ca-cert\") pod \"18d0c0ae-111d-4f92-ac13-58201176aa4c\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") "
Oct 29 22:41:31.193715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.193663    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-envoy\") pod \"18d0c0ae-111d-4f92-ac13-58201176aa4c\" (UID: \"18d0c0ae-111d-4f92-ac13-58201176aa4c\") "
Oct 29 22:41:31.193945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.193779    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-workload-socket" pod "18d0c0ae-111d-4f92-ac13-58201176aa4c" (UID: "18d0c0ae-111d-4f92-ac13-58201176aa4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.193945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.193867    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-workload-certs" pod "18d0c0ae-111d-4f92-ac13-58201176aa4c" (UID: "18d0c0ae-111d-4f92-ac13-58201176aa4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.194309 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.194284    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d0c0ae-111d-4f92-ac13-58201176aa4c-istiod-ca-cert" pod "18d0c0ae-111d-4f92-ac13-58201176aa4c" (UID: "18d0c0ae-111d-4f92-ac13-58201176aa4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:41:31.194857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.194833    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.194958 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.194863    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.194958 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.194877    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h9dw\" (UniqueName: \"kubernetes.io/projected/4042e16a-2f51-4588-b54c-bfce92dd57cc-kube-api-access-2h9dw\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.194958 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.194911    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.194958 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.194928    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.194958 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.194941    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.194958 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.194953    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d0c0ae-111d-4f92-ac13-58201176aa4c-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.195274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.194966    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.195274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.194999    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.195274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.195013    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.195274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.195025    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.195274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.195038    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4042e16a-2f51-4588-b54c-bfce92dd57cc-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.200461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.200430    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-envoy" pod "18d0c0ae-111d-4f92-ac13-58201176aa4c" (UID: "18d0c0ae-111d-4f92-ac13-58201176aa4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.200567 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.200541    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-podinfo" pod "18d0c0ae-111d-4f92-ac13-58201176aa4c" (UID: "18d0c0ae-111d-4f92-ac13-58201176aa4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:41:31.203090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.203052    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-token" pod "18d0c0ae-111d-4f92-ac13-58201176aa4c" (UID: "18d0c0ae-111d-4f92-ac13-58201176aa4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:31.203309 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.203284    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d0c0ae-111d-4f92-ac13-58201176aa4c-kube-api-access-vpg2b" pod "18d0c0ae-111d-4f92-ac13-58201176aa4c" (UID: "18d0c0ae-111d-4f92-ac13-58201176aa4c"). InnerVolumeSpecName "kube-api-access-vpg2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:31.206786 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.206763    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"]
Oct 29 22:41:31.212393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.212368    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3d1777-openshift-default-56f984b757-knrnv"]
Oct 29 22:41:31.213576 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.213558    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:41:31.229753 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.229713    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d1bb1ea-189a-4033-b702-7fba8db86cdf" path="/var/lib/kubelet/pods/5d1bb1ea-189a-4033-b702-7fba8db86cdf/volumes"
Oct 29 22:41:31.245426 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.245395    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"]
Oct 29 22:41:31.250975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.250948    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-wwxlh"]
Oct 29 22:41:31.295782 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.295741    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724aa157-d061-4f4d-8bcd-72d474e7207d-kube-api-access-22tlv\" (UniqueName: \"kubernetes.io/projected/724aa157-d061-4f4d-8bcd-72d474e7207d-kube-api-access-22tlv\") pod \"724aa157-d061-4f4d-8bcd-72d474e7207d\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") "
Oct 29 22:41:31.296004 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.295800    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-workload-socket\") pod \"724aa157-d061-4f4d-8bcd-72d474e7207d\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") "
Oct 29 22:41:31.296004 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.295934    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-data\") pod \"724aa157-d061-4f4d-8bcd-72d474e7207d\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") "
Oct 29 22:41:31.296004 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.295977    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-envoy\") pod \"724aa157-d061-4f4d-8bcd-72d474e7207d\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") "
Oct 29 22:41:31.296170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.296109    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-workload-certs\") pod \"724aa157-d061-4f4d-8bcd-72d474e7207d\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") "
Oct 29 22:41:31.296170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.296160    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/724aa157-d061-4f4d-8bcd-72d474e7207d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/724aa157-d061-4f4d-8bcd-72d474e7207d-istiod-ca-cert\") pod \"724aa157-d061-4f4d-8bcd-72d474e7207d\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") "
Oct 29 22:41:31.296309 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.296287    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-credential-socket\") pod \"724aa157-d061-4f4d-8bcd-72d474e7207d\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") "
Oct 29 22:41:31.296364 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.296353    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-token\" (UniqueName: \"kubernetes.io/projected/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-token\") pod \"724aa157-d061-4f4d-8bcd-72d474e7207d\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") "
Oct 29 22:41:31.296517 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.296499    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-podinfo\") pod \"724aa157-d061-4f4d-8bcd-72d474e7207d\" (UID: \"724aa157-d061-4f4d-8bcd-72d474e7207d\") "
Oct 29 22:41:31.297318 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.297290    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.297434 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.297325    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.297434 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.297340    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpg2b\" (UniqueName: \"kubernetes.io/projected/18d0c0ae-111d-4f92-ac13-58201176aa4c-kube-api-access-vpg2b\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.297434 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.297354    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d0c0ae-111d-4f92-ac13-58201176aa4c-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.298369 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.298139    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724aa157-d061-4f4d-8bcd-72d474e7207d-istiod-ca-cert" pod "724aa157-d061-4f4d-8bcd-72d474e7207d" (UID: "724aa157-d061-4f4d-8bcd-72d474e7207d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:41:31.298972 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.298948    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-credential-socket" pod "724aa157-d061-4f4d-8bcd-72d474e7207d" (UID: "724aa157-d061-4f4d-8bcd-72d474e7207d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.299246 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.299213    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-data" pod "724aa157-d061-4f4d-8bcd-72d474e7207d" (UID: "724aa157-d061-4f4d-8bcd-72d474e7207d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.299344 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.299143    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-workload-certs" pod "724aa157-d061-4f4d-8bcd-72d474e7207d" (UID: "724aa157-d061-4f4d-8bcd-72d474e7207d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.302033 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.301408    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-envoy" pod "724aa157-d061-4f4d-8bcd-72d474e7207d" (UID: "724aa157-d061-4f4d-8bcd-72d474e7207d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.302033 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.301416    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-podinfo" pod "724aa157-d061-4f4d-8bcd-72d474e7207d" (UID: "724aa157-d061-4f4d-8bcd-72d474e7207d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:41:31.302033 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.301509    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-workload-socket" pod "724aa157-d061-4f4d-8bcd-72d474e7207d" (UID: "724aa157-d061-4f4d-8bcd-72d474e7207d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.302033 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.301754    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-token" pod "724aa157-d061-4f4d-8bcd-72d474e7207d" (UID: "724aa157-d061-4f4d-8bcd-72d474e7207d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:31.302986 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.302960    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724aa157-d061-4f4d-8bcd-72d474e7207d-kube-api-access-22tlv" pod "724aa157-d061-4f4d-8bcd-72d474e7207d" (UID: "724aa157-d061-4f4d-8bcd-72d474e7207d"). InnerVolumeSpecName "kube-api-access-22tlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:31.399028 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.398995    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.399028 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.399025    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.399028 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.399034    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.399248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.399043    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22tlv\" (UniqueName: \"kubernetes.io/projected/724aa157-d061-4f4d-8bcd-72d474e7207d-kube-api-access-22tlv\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.399248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.399054    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.399248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.399063    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.399248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.399072    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.399248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.399079    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/724aa157-d061-4f4d-8bcd-72d474e7207d-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.399248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:31.399086    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/724aa157-d061-4f4d-8bcd-72d474e7207d-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:41:32.047625 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.047589    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"]
Oct 29 22:41:32.048391 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.048374    4559 cpu_manager.go:414] "containerMap: removing container" podUID="18d0c0ae-111d-4f92-ac13-58201176aa4c" containerName="istio-proxy"
Oct 29 22:41:32.048442 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.048394    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d0c0ae-111d-4f92-ac13-58201176aa4c" containerName="istio-proxy"
Oct 29 22:41:32.048442 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.048427    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4042e16a-2f51-4588-b54c-bfce92dd57cc" containerName="istio-proxy"
Oct 29 22:41:32.048442 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.048432    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4042e16a-2f51-4588-b54c-bfce92dd57cc" containerName="istio-proxy"
Oct 29 22:41:32.048555 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.048450    4559 cpu_manager.go:414] "containerMap: removing container" podUID="5d1bb1ea-189a-4033-b702-7fba8db86cdf" containerName="istio-proxy"
Oct 29 22:41:32.048555 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.048456    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1bb1ea-189a-4033-b702-7fba8db86cdf" containerName="istio-proxy"
Oct 29 22:41:32.048555 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.048465    4559 cpu_manager.go:414] "containerMap: removing container" podUID="724aa157-d061-4f4d-8bcd-72d474e7207d" containerName="istio-proxy"
Oct 29 22:41:32.048555 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.048470    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724aa157-d061-4f4d-8bcd-72d474e7207d" containerName="istio-proxy"
Oct 29 22:41:32.048555 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.048484    4559 cpu_manager.go:414] "containerMap: removing container" podUID="5d1bb1ea-189a-4033-b702-7fba8db86cdf" containerName="istio-proxy"
Oct 29 22:41:32.048555 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.048488    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1bb1ea-189a-4033-b702-7fba8db86cdf" containerName="istio-proxy"
Oct 29 22:41:32.048758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.048599    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4042e16a-2f51-4588-b54c-bfce92dd57cc" containerName="istio-proxy"
Oct 29 22:41:32.048758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.048612    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1bb1ea-189a-4033-b702-7fba8db86cdf" containerName="istio-proxy"
Oct 29 22:41:32.048758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.048619    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="724aa157-d061-4f4d-8bcd-72d474e7207d" containerName="istio-proxy"
Oct 29 22:41:32.048758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.048627    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1bb1ea-189a-4033-b702-7fba8db86cdf" containerName="istio-proxy"
Oct 29 22:41:32.048758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.048633    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d0c0ae-111d-4f92-ac13-58201176aa4c" containerName="istio-proxy"
Oct 29 22:41:32.053700 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.053677    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"
Oct 29 22:41:32.085838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.077245    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c7d192-dockercfg-rqwjd\""
Oct 29 22:41:32.101535 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.101493    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"]
Oct 29 22:41:32.145835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.138417    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/068d019b-57d0-4974-9aba-aab57e23b4fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp\" (UID: \"068d019b-57d0-4974-9aba-aab57e23b4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"
Oct 29 22:41:32.147076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.146389    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c7d192\""
Oct 29 22:41:32.147612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.147583    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2npj\" (UniqueName: \"kubernetes.io/projected/068d019b-57d0-4974-9aba-aab57e23b4fb-kube-api-access-j2npj\") pod \"payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp\" (UID: \"068d019b-57d0-4974-9aba-aab57e23b4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"
Oct 29 22:41:32.172432 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.172391    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"]
Oct 29 22:41:32.211391 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.211360    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"
Oct 29 22:41:32.250835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.249303    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2npj\" (UniqueName: \"kubernetes.io/projected/068d019b-57d0-4974-9aba-aab57e23b4fb-kube-api-access-j2npj\") pod \"payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp\" (UID: \"068d019b-57d0-4974-9aba-aab57e23b4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"
Oct 29 22:41:32.250835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.249493    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/068d019b-57d0-4974-9aba-aab57e23b4fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp\" (UID: \"068d019b-57d0-4974-9aba-aab57e23b4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"
Oct 29 22:41:32.250835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.250549    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/068d019b-57d0-4974-9aba-aab57e23b4fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp\" (UID: \"068d019b-57d0-4974-9aba-aab57e23b4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"
Oct 29 22:41:32.338149 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.338033    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"]
Oct 29 22:41:32.338925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.338883    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"
Oct 29 22:41:32.339471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.339435    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8" event={"ID":"18d0c0ae-111d-4f92-ac13-58201176aa4c","Type":"ContainerDied","Data":"ac29b6be8fe1c1b1dfda60e6ec7943d1a5ef98575883747eb7f5351f60bb0423"}
Oct 29 22:41:32.339664 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.339638    4559 scope.go:122] "RemoveContainer" containerID="e4e583091db7b0c2965c0f888d4d74a512706f89f013bb25df98416083cb5a85"
Oct 29 22:41:32.398855 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.381562    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4" event={"ID":"724aa157-d061-4f4d-8bcd-72d474e7207d","Type":"ContainerDied","Data":"a33801e220ff0a2cf252d07644dc61ae04d191484da9bcfc1819b0165efbd970"}
Oct 29 22:41:32.398855 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.381721    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"
Oct 29 22:41:32.463551 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.463501    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s86d2\" (UniqueName: \"kubernetes.io/projected/5a29c9ee-f1a8-460a-a4b7-25625aa72bfd-kube-api-access-s86d2\") pod \"payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm\" (UID: \"5a29c9ee-f1a8-460a-a4b7-25625aa72bfd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"
Oct 29 22:41:32.463952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.463927    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a29c9ee-f1a8-460a-a4b7-25625aa72bfd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm\" (UID: \"5a29c9ee-f1a8-460a-a4b7-25625aa72bfd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"
Oct 29 22:41:32.478934 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.478775    4559 scope.go:122] "RemoveContainer" containerID="65879f6d2226944f10890f9ed3492b76361708fcb579eb30dec0e317e97d5d90"
Oct 29 22:41:32.499936 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.499905    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"]
Oct 29 22:41:32.504461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.504398    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2npj\" (UniqueName: \"kubernetes.io/projected/068d019b-57d0-4974-9aba-aab57e23b4fb-kube-api-access-j2npj\") pod \"payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp\" (UID: \"068d019b-57d0-4974-9aba-aab57e23b4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"
Oct 29 22:41:32.506343 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.506312    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-zqtg8"]
Oct 29 22:41:32.535055 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.534977    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"]
Oct 29 22:41:32.539732 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.539698    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-bmbm4"]
Oct 29 22:41:32.565773 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.565731    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a29c9ee-f1a8-460a-a4b7-25625aa72bfd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm\" (UID: \"5a29c9ee-f1a8-460a-a4b7-25625aa72bfd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"
Oct 29 22:41:32.565993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.565902    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s86d2\" (UniqueName: \"kubernetes.io/projected/5a29c9ee-f1a8-460a-a4b7-25625aa72bfd-kube-api-access-s86d2\") pod \"payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm\" (UID: \"5a29c9ee-f1a8-460a-a4b7-25625aa72bfd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"
Oct 29 22:41:32.566658 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.566629    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a29c9ee-f1a8-460a-a4b7-25625aa72bfd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm\" (UID: \"5a29c9ee-f1a8-460a-a4b7-25625aa72bfd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"
Oct 29 22:41:32.587794 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.587756    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s86d2\" (UniqueName: \"kubernetes.io/projected/5a29c9ee-f1a8-460a-a4b7-25625aa72bfd-kube-api-access-s86d2\") pod \"payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm\" (UID: \"5a29c9ee-f1a8-460a-a4b7-25625aa72bfd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"
Oct 29 22:41:32.633048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.632955    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:32.651983 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.651954    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:32.697763 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.697732    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"
Oct 29 22:41:32.701898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:32.701870    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"
Oct 29 22:41:33.044549 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:41:33.044520    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a29c9ee_f1a8_460a_a4b7_25625aa72bfd.slice/crio-53ec7efb9ab631ed020158adb1c878fa22340e05380205f107f961f11273d3e7 WatchSource:0}: Error finding container 53ec7efb9ab631ed020158adb1c878fa22340e05380205f107f961f11273d3e7: Status 404 returned error can't find the container with id 53ec7efb9ab631ed020158adb1c878fa22340e05380205f107f961f11273d3e7
Oct 29 22:41:33.045946 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:33.045912    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"]
Oct 29 22:41:33.078360 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:33.074629    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"]
Oct 29 22:41:33.079655 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:41:33.079621    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068d019b_57d0_4974_9aba_aab57e23b4fb.slice/crio-4d71c4f18a394a21ba38b8634257f28a5636059099ab37e65488e36a685c494f WatchSource:0}: Error finding container 4d71c4f18a394a21ba38b8634257f28a5636059099ab37e65488e36a685c494f: Status 404 returned error can't find the container with id 4d71c4f18a394a21ba38b8634257f28a5636059099ab37e65488e36a685c494f
Oct 29 22:41:33.230718 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:33.230687    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d0c0ae-111d-4f92-ac13-58201176aa4c" path="/var/lib/kubelet/pods/18d0c0ae-111d-4f92-ac13-58201176aa4c/volumes"
Oct 29 22:41:33.231251 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:33.231231    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4042e16a-2f51-4588-b54c-bfce92dd57cc" path="/var/lib/kubelet/pods/4042e16a-2f51-4588-b54c-bfce92dd57cc/volumes"
Oct 29 22:41:33.231682 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:33.231668    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="724aa157-d061-4f4d-8bcd-72d474e7207d" path="/var/lib/kubelet/pods/724aa157-d061-4f4d-8bcd-72d474e7207d/volumes"
Oct 29 22:41:33.523835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:33.523472    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm" event={"ID":"5a29c9ee-f1a8-460a-a4b7-25625aa72bfd","Type":"ContainerStarted","Data":"6a37d4d91e464cb50096a2f5838e5d8b7d9103645533e9473d3b8dd55f101c59"}
Oct 29 22:41:33.523835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:33.523591    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm" event={"ID":"5a29c9ee-f1a8-460a-a4b7-25625aa72bfd","Type":"ContainerStarted","Data":"53ec7efb9ab631ed020158adb1c878fa22340e05380205f107f961f11273d3e7"}
Oct 29 22:41:33.531839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:33.525093    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"
Oct 29 22:41:33.598747 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:33.598603    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm" podStartSLOduration=1.598579421 podStartE2EDuration="1.598579421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:33.586325255 +0000 UTC m=+1632.497768808" watchObservedRunningTime="2026-10-29 22:41:33.598579421 +0000 UTC m=+1632.510022977"
Oct 29 22:41:33.612695 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:33.601852    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp" event={"ID":"068d019b-57d0-4974-9aba-aab57e23b4fb","Type":"ContainerStarted","Data":"4c1d88065c83286b5a0dc5ba5c747bdc3922fca7030a10c6c96edf690d0a42cd"}
Oct 29 22:41:33.612695 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:33.612328    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"
Oct 29 22:41:33.612695 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:33.612362    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp" event={"ID":"068d019b-57d0-4974-9aba-aab57e23b4fb","Type":"ContainerStarted","Data":"4d71c4f18a394a21ba38b8634257f28a5636059099ab37e65488e36a685c494f"}
Oct 29 22:41:33.631860 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:33.631482    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:41:33.652205 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:33.652146    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp" podStartSLOduration=1.6521272919999999 podStartE2EDuration="1.652127292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:33.650894982 +0000 UTC m=+1632.562338537" watchObservedRunningTime="2026-10-29 22:41:33.652127292 +0000 UTC m=+1632.563570847"
Oct 29 22:41:38.375228 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:38.375191    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:38.388140 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:38.388112    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:38.702463 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:38.702428    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:41:39.775297 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:39.773882    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"]
Oct 29 22:41:39.780850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:39.780525    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:39.797465 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:39.797439    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5a69c9-openshift-default-dockercfg-w8dpp\""
Oct 29 22:41:39.824887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:39.824851    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"]
Oct 29 22:41:39.942836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:39.892590    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-workload-socket\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:39.942836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:39.892672    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpxbv\" (UniqueName: \"kubernetes.io/projected/b6cffbad-4429-4e61-b582-5b45fda07b48-kube-api-access-gpxbv\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:39.942836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:39.892754    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-envoy\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:39.942836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:39.892792    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-workload-certs\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:39.942836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:39.892849    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-token\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:39.942836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:39.892955    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-data\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:39.942836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:39.893041    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-credential-socket\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:39.942836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:39.893068    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-podinfo\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:39.942836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:39.893267    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cffbad-4429-4e61-b582-5b45fda07b48-istiod-ca-cert\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.124894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.107646    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cffbad-4429-4e61-b582-5b45fda07b48-istiod-ca-cert\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.124894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.107737    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-workload-socket\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.124894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.107794    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpxbv\" (UniqueName: \"kubernetes.io/projected/b6cffbad-4429-4e61-b582-5b45fda07b48-kube-api-access-gpxbv\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.124894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.107901    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-envoy\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.124894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.107943    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-workload-certs\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.124894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.107981    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-token\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.124894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.108093    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-data\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.124894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.108164    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-credential-socket\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.124894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.108194    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-podinfo\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.146900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.146862    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cffbad-4429-4e61-b582-5b45fda07b48-istiod-ca-cert\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.148945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.148896    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-workload-certs\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.149413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.149372    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-workload-socket\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.149801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.149768    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-data\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.150723 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.150670    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-credential-socket\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.271428 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.268727    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-envoy\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.328494 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.328459    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-podinfo\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.330029 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.329389    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-token\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.335391 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.334761    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpxbv\" (UniqueName: \"kubernetes.io/projected/b6cffbad-4429-4e61-b582-5b45fda07b48-kube-api-access-gpxbv\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.397379 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.397343    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:40.724504 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:41:40.724467    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cffbad_4429_4e61_b582_5b45fda07b48.slice/crio-b6c9bc96d5a0b793fc7325bc8b79a46c366dc43d2b3c15fe06879a3b91671da7 WatchSource:0}: Error finding container b6c9bc96d5a0b793fc7325bc8b79a46c366dc43d2b3c15fe06879a3b91671da7: Status 404 returned error can't find the container with id b6c9bc96d5a0b793fc7325bc8b79a46c366dc43d2b3c15fe06879a3b91671da7
Oct 29 22:41:40.724704 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.724685    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"]
Oct 29 22:41:40.727480 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.727445    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:41:40.727589 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.727525    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:41:40.727589 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:40.727555    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:41:41.714983 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:41.714938    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh" event={"ID":"b6cffbad-4429-4e61-b582-5b45fda07b48","Type":"ContainerStarted","Data":"11457e1ec66e14af1203fc2fe0d0bd584389b64c10606d5266cef0f8a5f31ffb"}
Oct 29 22:41:41.715475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:41.715013    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh" event={"ID":"b6cffbad-4429-4e61-b582-5b45fda07b48","Type":"ContainerStarted","Data":"b6c9bc96d5a0b793fc7325bc8b79a46c366dc43d2b3c15fe06879a3b91671da7"}
Oct 29 22:41:41.773676 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:41.773595    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh" podStartSLOduration=2.773573107 podStartE2EDuration="2.773573107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:41.767727657 +0000 UTC m=+1640.679171213" watchObservedRunningTime="2026-10-29 22:41:41.773573107 +0000 UTC m=+1640.685016662"
Oct 29 22:41:42.397789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:42.397757    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:42.406986 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:42.406721    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:42.722821 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:42.722726    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:42.723896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:42.723863    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:41:43.980603 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:41:43.963595    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:41:44.663319 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:44.663284    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"
Oct 29 22:41:45.670830 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:45.670252    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"
Oct 29 22:41:46.322838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.309867    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"]
Oct 29 22:41:46.399135 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.399102    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"]
Oct 29 22:41:46.399527 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.399503    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"
Oct 29 22:41:46.438227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.435523    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5a69c9-dockercfg-dlddn\""
Oct 29 22:41:46.442564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.442533    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"]
Oct 29 22:41:46.447226 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.447199    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5a69c9\""
Oct 29 22:41:46.448594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.448570    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"]
Oct 29 22:41:46.448891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.448804    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:41:46.490465 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.490440    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f5b015-9b45-407f-ae2b-9d9645b5f09b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq\" (UID: \"a5f5b015-9b45-407f-ae2b-9d9645b5f09b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"
Oct 29 22:41:46.490831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.490796    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg5h5\" (UniqueName: \"kubernetes.io/projected/a5f5b015-9b45-407f-ae2b-9d9645b5f09b-kube-api-access-sg5h5\") pod \"payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq\" (UID: \"a5f5b015-9b45-407f-ae2b-9d9645b5f09b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"
Oct 29 22:41:46.545041 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.544987    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fee294-serving-cert\""
Oct 29 22:41:46.622882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.618727    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h227s\" (UniqueName: \"kubernetes.io/projected/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-kube-api-access-h227s\") pod \"maas-api-e2e-inf-a-fee294-5468577b6-dn5qd\" (UID: \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:41:46.622882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.618889    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-metrics-certs\") pod \"maas-api-e2e-inf-a-fee294-5468577b6-dn5qd\" (UID: \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:41:46.622882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.618986    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg5h5\" (UniqueName: \"kubernetes.io/projected/a5f5b015-9b45-407f-ae2b-9d9645b5f09b-kube-api-access-sg5h5\") pod \"payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq\" (UID: \"a5f5b015-9b45-407f-ae2b-9d9645b5f09b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"
Oct 29 22:41:46.622882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.619058    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-maas-api-tls\") pod \"maas-api-e2e-inf-a-fee294-5468577b6-dn5qd\" (UID: \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:41:46.622882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.619143    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fee294-5468577b6-dn5qd\" (UID: \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:41:46.622882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.619286    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f5b015-9b45-407f-ae2b-9d9645b5f09b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq\" (UID: \"a5f5b015-9b45-407f-ae2b-9d9645b5f09b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"
Oct 29 22:41:46.630504 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.630462    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f5b015-9b45-407f-ae2b-9d9645b5f09b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq\" (UID: \"a5f5b015-9b45-407f-ae2b-9d9645b5f09b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"
Oct 29 22:41:46.670727 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.670698    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg5h5\" (UniqueName: \"kubernetes.io/projected/a5f5b015-9b45-407f-ae2b-9d9645b5f09b-kube-api-access-sg5h5\") pod \"payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq\" (UID: \"a5f5b015-9b45-407f-ae2b-9d9645b5f09b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"
Oct 29 22:41:46.721094 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.721059    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h227s\" (UniqueName: \"kubernetes.io/projected/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-kube-api-access-h227s\") pod \"maas-api-e2e-inf-a-fee294-5468577b6-dn5qd\" (UID: \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:41:46.721589 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.721159    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-metrics-certs\") pod \"maas-api-e2e-inf-a-fee294-5468577b6-dn5qd\" (UID: \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:41:46.721589 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.721249    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-maas-api-tls\") pod \"maas-api-e2e-inf-a-fee294-5468577b6-dn5qd\" (UID: \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:41:46.721589 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.721306    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fee294-5468577b6-dn5qd\" (UID: \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:41:46.722174 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.722144    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fee294-5468577b6-dn5qd\" (UID: \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:41:46.724123 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.724081    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-maas-api-tls\") pod \"maas-api-e2e-inf-a-fee294-5468577b6-dn5qd\" (UID: \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:41:46.724539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.724511    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-metrics-certs\") pod \"maas-api-e2e-inf-a-fee294-5468577b6-dn5qd\" (UID: \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:41:46.746075 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.746040    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h227s\" (UniqueName: \"kubernetes.io/projected/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-kube-api-access-h227s\") pod \"maas-api-e2e-inf-a-fee294-5468577b6-dn5qd\" (UID: \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:41:46.793304 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.793267    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"
Oct 29 22:41:46.799871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:46.799795    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:41:47.155719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:47.145549    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"]
Oct 29 22:41:47.172564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:47.171256    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"]
Oct 29 22:41:47.941894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:47.938659    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq" event={"ID":"a5f5b015-9b45-407f-ae2b-9d9645b5f09b","Type":"ContainerStarted","Data":"9ea5448f1da7cee51107198c0864e68bae1e77652b15d621f0599d28ac0071b7"}
Oct 29 22:41:47.941894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:47.938729    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq" event={"ID":"a5f5b015-9b45-407f-ae2b-9d9645b5f09b","Type":"ContainerStarted","Data":"e622ec996ed6ee70047e9ea7e31cf1448e932d232b88540e175e1300765469d3"}
Oct 29 22:41:47.955930 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:47.951901    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"
Oct 29 22:41:47.955930 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:47.954529    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd" event={"ID":"ca9a1393-ade2-4ee7-9dd6-a4818740a29c","Type":"ContainerStarted","Data":"73557eff956360e8adc519c3ca503c32220041b501a8f24dd2a91c280d004da2"}
Oct 29 22:41:47.955930 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:47.954585    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd" event={"ID":"ca9a1393-ade2-4ee7-9dd6-a4818740a29c","Type":"ContainerStarted","Data":"8d4b87707cfea85f87244efffde49240dda0cf60d9a00b3ea0f6e2a8ea4b8ac0"}
Oct 29 22:41:47.955930 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:47.955533    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:41:48.068208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:48.067086    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq" podStartSLOduration=2.0670654920000002 podStartE2EDuration="2.067065492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:48.04444434 +0000 UTC m=+1646.955887921" watchObservedRunningTime="2026-10-29 22:41:48.067065492 +0000 UTC m=+1646.978509047"
Oct 29 22:41:49.898554 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:49.898502    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd" podStartSLOduration=3.542575641 podStartE2EDuration="3.898487711s" totalImagesPullingTime="355.91207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:48.104433183 +0000 UTC m=+1647.015876736" watchObservedRunningTime="2026-10-29 22:41:49.898487711 +0000 UTC m=+1648.809931264"
Oct 29 22:41:49.899724 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:49.899699    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"]
Oct 29 22:41:49.904654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:49.904626    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"
Oct 29 22:41:49.928326 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:49.925289    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-9aafe6\""
Oct 29 22:41:49.952836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:49.948089    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-9aafe6-dockercfg-fwzlb\""
Oct 29 22:41:50.071832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:50.050165    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lkch\" (UniqueName: \"kubernetes.io/projected/299a03af-cb40-46b1-b42d-8afa20c7cd00-kube-api-access-8lkch\") pod \"payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h\" (UID: \"299a03af-cb40-46b1-b42d-8afa20c7cd00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"
Oct 29 22:41:50.071832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:50.050301    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299a03af-cb40-46b1-b42d-8afa20c7cd00-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h\" (UID: \"299a03af-cb40-46b1-b42d-8afa20c7cd00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"
Oct 29 22:41:50.149928 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:50.148877    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"]
Oct 29 22:41:50.249951 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:50.249900    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"]
Oct 29 22:41:50.279201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:50.251192    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lkch\" (UniqueName: \"kubernetes.io/projected/299a03af-cb40-46b1-b42d-8afa20c7cd00-kube-api-access-8lkch\") pod \"payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h\" (UID: \"299a03af-cb40-46b1-b42d-8afa20c7cd00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"
Oct 29 22:41:50.280090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:50.279491    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299a03af-cb40-46b1-b42d-8afa20c7cd00-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h\" (UID: \"299a03af-cb40-46b1-b42d-8afa20c7cd00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"
Oct 29 22:41:50.280591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:50.280514    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299a03af-cb40-46b1-b42d-8afa20c7cd00-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h\" (UID: \"299a03af-cb40-46b1-b42d-8afa20c7cd00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"
Oct 29 22:41:50.390670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:50.390639    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"
Oct 29 22:41:50.393431 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:50.393400    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"]
Oct 29 22:41:50.394884 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:50.394859    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lkch\" (UniqueName: \"kubernetes.io/projected/299a03af-cb40-46b1-b42d-8afa20c7cd00-kube-api-access-8lkch\") pod \"payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h\" (UID: \"299a03af-cb40-46b1-b42d-8afa20c7cd00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"
Oct 29 22:41:50.589244 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:50.589179    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341bcfef-7b8c-4732-a60c-2849614df3b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr\" (UID: \"341bcfef-7b8c-4732-a60c-2849614df3b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"
Oct 29 22:41:50.589464 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:50.589441    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmfvk\" (UniqueName: \"kubernetes.io/projected/341bcfef-7b8c-4732-a60c-2849614df3b7-kube-api-access-gmfvk\") pod \"payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr\" (UID: \"341bcfef-7b8c-4732-a60c-2849614df3b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"
Oct 29 22:41:50.698888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:50.683105    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"
Oct 29 22:41:50.698888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:50.693370    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341bcfef-7b8c-4732-a60c-2849614df3b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr\" (UID: \"341bcfef-7b8c-4732-a60c-2849614df3b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"
Oct 29 22:41:50.698888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:50.693505    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmfvk\" (UniqueName: \"kubernetes.io/projected/341bcfef-7b8c-4732-a60c-2849614df3b7-kube-api-access-gmfvk\") pod \"payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr\" (UID: \"341bcfef-7b8c-4732-a60c-2849614df3b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"
Oct 29 22:41:50.698888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:50.694738    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341bcfef-7b8c-4732-a60c-2849614df3b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr\" (UID: \"341bcfef-7b8c-4732-a60c-2849614df3b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"
Oct 29 22:41:50.746411 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:50.746329    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmfvk\" (UniqueName: \"kubernetes.io/projected/341bcfef-7b8c-4732-a60c-2849614df3b7-kube-api-access-gmfvk\") pod \"payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr\" (UID: \"341bcfef-7b8c-4732-a60c-2849614df3b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"
Oct 29 22:41:51.022627 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:51.022537    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"
Oct 29 22:41:51.076451 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:41:51.076408    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299a03af_cb40_46b1_b42d_8afa20c7cd00.slice/crio-de3e9e3df99f8c004e94ea154a04030f7358ec6a777c7cbfefe29a78c586b2d5 WatchSource:0}: Error finding container de3e9e3df99f8c004e94ea154a04030f7358ec6a777c7cbfefe29a78c586b2d5: Status 404 returned error can't find the container with id de3e9e3df99f8c004e94ea154a04030f7358ec6a777c7cbfefe29a78c586b2d5
Oct 29 22:41:51.077087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:51.077061    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"]
Oct 29 22:41:51.341757 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:51.341724    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"]
Oct 29 22:41:51.342337 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:41:51.342312    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341bcfef_7b8c_4732_a60c_2849614df3b7.slice/crio-d2a59da1b908c89f5e8828c46e3d1f3dac984e6707099cefb1f5edb5b3d9b33a WatchSource:0}: Error finding container d2a59da1b908c89f5e8828c46e3d1f3dac984e6707099cefb1f5edb5b3d9b33a: Status 404 returned error can't find the container with id d2a59da1b908c89f5e8828c46e3d1f3dac984e6707099cefb1f5edb5b3d9b33a
Oct 29 22:41:51.388756 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:51.388726    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h" event={"ID":"299a03af-cb40-46b1-b42d-8afa20c7cd00","Type":"ContainerStarted","Data":"f27e188cea94f371d194cfe0a0111d1ef6f7a5f44fa93061767d9fefd38867ae"}
Oct 29 22:41:51.388905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:51.388778    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h" event={"ID":"299a03af-cb40-46b1-b42d-8afa20c7cd00","Type":"ContainerStarted","Data":"de3e9e3df99f8c004e94ea154a04030f7358ec6a777c7cbfefe29a78c586b2d5"}
Oct 29 22:41:51.389390 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:51.389368    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"
Oct 29 22:41:51.391185 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:51.391161    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr" event={"ID":"341bcfef-7b8c-4732-a60c-2849614df3b7","Type":"ContainerStarted","Data":"d2a59da1b908c89f5e8828c46e3d1f3dac984e6707099cefb1f5edb5b3d9b33a"}
Oct 29 22:41:51.435233 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:51.435182    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h" podStartSLOduration=2.435168742 podStartE2EDuration="2.435168742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:51.432217807 +0000 UTC m=+1650.343661373" watchObservedRunningTime="2026-10-29 22:41:51.435168742 +0000 UTC m=+1650.346612296"
Oct 29 22:41:52.480063 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:52.480023    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr" event={"ID":"341bcfef-7b8c-4732-a60c-2849614df3b7","Type":"ContainerStarted","Data":"86e6ce7b87b79adf712b19c0b2f6885ef9b5cf53e0028e85423f5ea71ac8ed15"}
Oct 29 22:41:52.480656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:52.480626    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"
Oct 29 22:41:52.522925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:52.522860    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr" podStartSLOduration=3.522845001 podStartE2EDuration="3.522845001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:52.520024591 +0000 UTC m=+1651.431468145" watchObservedRunningTime="2026-10-29 22:41:52.522845001 +0000 UTC m=+1651.434288554"
Oct 29 22:41:55.385758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:41:55.385729    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:42:00.281376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:00.281337    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:42:00.382881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:00.382851    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"
Oct 29 22:42:02.502886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:02.502012    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"
Oct 29 22:42:04.553835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:04.553001    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"
Oct 29 22:42:13.701187 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:42:13.701143    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:42:21.094299 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.094253    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"]
Oct 29 22:42:21.099478 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.099453    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 22:42:21.114722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.114691    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2\" (UID: \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 22:42:21.114959 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.114935    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2\" (UID: \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 22:42:21.115102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.115045    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2\" (UID: \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 22:42:21.115102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.115072    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg7xq\" (UniqueName: \"kubernetes.io/projected/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-kube-api-access-tg7xq\") pod \"maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2\" (UID: \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 22:42:21.136877 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.136839    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"]
Oct 29 22:42:21.137655 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.137630    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-9aafe6-serving-cert\""
Oct 29 22:42:21.217001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.216958    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2\" (UID: \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 22:42:21.217189 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.217083    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2\" (UID: \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 22:42:21.217189 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.217113    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg7xq\" (UniqueName: \"kubernetes.io/projected/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-kube-api-access-tg7xq\") pod \"maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2\" (UID: \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 22:42:21.217269 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.217225    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2\" (UID: \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 22:42:21.217722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.217691    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2\" (UID: \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 22:42:21.219906 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.219879    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2\" (UID: \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 22:42:21.220091 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.220066    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2\" (UID: \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 22:42:21.240077 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.240048    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg7xq\" (UniqueName: \"kubernetes.io/projected/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-kube-api-access-tg7xq\") pod \"maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2\" (UID: \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 22:42:21.415937 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.415456    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 22:42:21.773389 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:42:21.773358    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1040d20_45e0_4df0_bc5a_2ca67e5e5108.slice/crio-a971c384e33ae22ce45bde55c6647a9365d838469cb81b295e0ee921a058a2b7 WatchSource:0}: Error finding container a971c384e33ae22ce45bde55c6647a9365d838469cb81b295e0ee921a058a2b7: Status 404 returned error can't find the container with id a971c384e33ae22ce45bde55c6647a9365d838469cb81b295e0ee921a058a2b7
Oct 29 22:42:21.774802 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.774773    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"]
Oct 29 22:42:21.819359 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:21.819319    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2" event={"ID":"a1040d20-45e0-4df0-bc5a-2ca67e5e5108","Type":"ContainerStarted","Data":"a971c384e33ae22ce45bde55c6647a9365d838469cb81b295e0ee921a058a2b7"}
Oct 29 22:42:22.829240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:22.829207    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2" event={"ID":"a1040d20-45e0-4df0-bc5a-2ca67e5e5108","Type":"ContainerStarted","Data":"6ca7c57a1b042aebfaae4b0fc22e934591411b859f3c7ae4b4e97956bc7237bd"}
Oct 29 22:42:22.829710 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:22.829281    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 22:42:27.243800 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.243738    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2" podStartSLOduration=5.9491585780000005 podStartE2EDuration="6.243717833s" totalImagesPullingTime="294.559255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:22.892364918 +0000 UTC m=+1681.803808479" watchObservedRunningTime="2026-10-29 22:42:27.243717833 +0000 UTC m=+1686.155161388"
Oct 29 22:42:27.245977 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.245936    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"]
Oct 29 22:42:27.253291 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.253268    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.281627 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.281593    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"]
Oct 29 22:42:27.287927 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.287898    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8147a8-openshift-default-dockercfg-hmb4l\""
Oct 29 22:42:27.314229 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.314195    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-workload-socket\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.314488 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.314246    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-token\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.314488 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.314374    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-envoy\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.314586 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.314498    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istiod-ca-cert\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.314622 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.314599    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-podinfo\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.314702 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.314680    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-credential-socket\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.314861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.314804    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n2vr\" (UniqueName: \"kubernetes.io/projected/44273e03-eb0c-48e0-82ed-9adfc7badd3d-kube-api-access-5n2vr\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.314953 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.314924    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-workload-certs\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.315015 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.314968    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-data\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.427833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.427105    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-token\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.427833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.427188    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-envoy\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.427833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.427314    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istiod-ca-cert\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.427833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.427430    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-podinfo\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.427833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.427529    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-credential-socket\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.427833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.427591    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n2vr\" (UniqueName: \"kubernetes.io/projected/44273e03-eb0c-48e0-82ed-9adfc7badd3d-kube-api-access-5n2vr\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.427833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.427668    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-workload-certs\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.427833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.427710    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-data\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.427833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.427792    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-workload-socket\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.428474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.428349    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-workload-socket\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.444833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.442772    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istiod-ca-cert\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.457873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.447316    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-credential-socket\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.528264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.449003    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-workload-certs\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.528264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.449230    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-data\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.528264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.495452    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-podinfo\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.528264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.498444    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-token\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.528264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.499086    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n2vr\" (UniqueName: \"kubernetes.io/projected/44273e03-eb0c-48e0-82ed-9adfc7badd3d-kube-api-access-5n2vr\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.528264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.513458    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-envoy\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:27.577837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:27.577028    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:28.152345 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:28.152307    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"]
Oct 29 22:42:28.158567 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:42:28.158524    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44273e03_eb0c_48e0_82ed_9adfc7badd3d.slice/crio-8b1d265e4ea903fab4037ad034d62bcc83f2270d093d86f45dc20b81b93e2eaf WatchSource:0}: Error finding container 8b1d265e4ea903fab4037ad034d62bcc83f2270d093d86f45dc20b81b93e2eaf: Status 404 returned error can't find the container with id 8b1d265e4ea903fab4037ad034d62bcc83f2270d093d86f45dc20b81b93e2eaf
Oct 29 22:42:28.162384 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:28.162333    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:42:28.162640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:28.162432    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:42:28.162640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:28.162482    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:42:28.844193 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:28.844163    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 22:42:28.884399 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:28.884358    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd" event={"ID":"44273e03-eb0c-48e0-82ed-9adfc7badd3d","Type":"ContainerStarted","Data":"305fff2921d2a8374ea20fbb65a2458789e31ffc891310c3c058f9df56884ab7"}
Oct 29 22:42:28.884607 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:28.884419    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd" event={"ID":"44273e03-eb0c-48e0-82ed-9adfc7badd3d","Type":"ContainerStarted","Data":"8b1d265e4ea903fab4037ad034d62bcc83f2270d093d86f45dc20b81b93e2eaf"}
Oct 29 22:42:28.931574 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:28.931497    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd" podStartSLOduration=1.931476559 podStartE2EDuration="1.931476559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:28.929311502 +0000 UTC m=+1687.840755077" watchObservedRunningTime="2026-10-29 22:42:28.931476559 +0000 UTC m=+1687.842920115"
Oct 29 22:42:29.577825 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:29.577776    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:29.586005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:29.585978    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:29.892406 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:29.892369    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:29.893425 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:29.893404    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:42:43.702164 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:42:43.702129    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:42:47.581860 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.581243    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"]
Oct 29 22:42:47.588769 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.588741    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.605804 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.605775    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2a3ef0-openshift-default-dockercfg-wm7cg\""
Oct 29 22:42:47.657973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.657903    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"]
Oct 29 22:42:47.684046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.684007    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-podinfo\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.684460 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.684430    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-envoy\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.684929 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.684904    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pcmq\" (UniqueName: \"kubernetes.io/projected/81d39dc4-cb6c-4265-bcb5-4356168496fb-kube-api-access-5pcmq\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.692074 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.690688    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-workload-certs\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.733951 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.733906    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d39dc4-cb6c-4265-bcb5-4356168496fb-istiod-ca-cert\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.792870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.744017    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-credential-socket\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.792870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.763904    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-token\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.792870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.763975    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-workload-socket\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.792870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.764182    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-data\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.890887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.886985    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-data\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.890887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.887134    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-podinfo\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.890887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.887189    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-envoy\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.890887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.887248    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pcmq\" (UniqueName: \"kubernetes.io/projected/81d39dc4-cb6c-4265-bcb5-4356168496fb-kube-api-access-5pcmq\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.890887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.887293    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-workload-certs\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.890887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.887316    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d39dc4-cb6c-4265-bcb5-4356168496fb-istiod-ca-cert\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.890887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.887430    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-credential-socket\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.890887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.887469    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-token\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.890887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.887522    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-workload-socket\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.890887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.890610    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-data\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.938839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.895187    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-workload-certs\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.938839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.895860    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d39dc4-cb6c-4265-bcb5-4356168496fb-istiod-ca-cert\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.938839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.896112    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-credential-socket\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.940413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.940379    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-workload-socket\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.981764 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.974436    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-podinfo\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:47.981764 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:47.975386    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-envoy\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:48.145799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:48.145627    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pcmq\" (UniqueName: \"kubernetes.io/projected/81d39dc4-cb6c-4265-bcb5-4356168496fb-kube-api-access-5pcmq\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:48.156858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:48.156146    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-token\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:48.217918 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:48.217888    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:48.898114 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:42:48.898081    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d39dc4_cb6c_4265_bcb5_4356168496fb.slice/crio-2ef62e357310c18baa18909f5e4b0ca243c2ccd564f3cf254e34b402a1147daa WatchSource:0}: Error finding container 2ef62e357310c18baa18909f5e4b0ca243c2ccd564f3cf254e34b402a1147daa: Status 404 returned error can't find the container with id 2ef62e357310c18baa18909f5e4b0ca243c2ccd564f3cf254e34b402a1147daa
Oct 29 22:42:48.899380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:48.899354    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"]
Oct 29 22:42:48.901261 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:48.901227    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:42:48.901320 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:48.901298    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:42:48.901371 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:48.901326    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:42:49.365630 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:49.365532    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6" event={"ID":"81d39dc4-cb6c-4265-bcb5-4356168496fb","Type":"ContainerStarted","Data":"50397d1c8c03c78af8db4d8ddc218aaca5002c73f22c9a41a7dc9665adccd445"}
Oct 29 22:42:49.366022 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:49.365976    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6" event={"ID":"81d39dc4-cb6c-4265-bcb5-4356168496fb","Type":"ContainerStarted","Data":"2ef62e357310c18baa18909f5e4b0ca243c2ccd564f3cf254e34b402a1147daa"}
Oct 29 22:42:50.224499 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:50.224457    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:50.232463 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:50.232429    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:50.296345 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:50.296282    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6" podStartSLOduration=3.296263291 podStartE2EDuration="3.296263291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:49.43718061 +0000 UTC m=+1708.348624165" watchObservedRunningTime="2026-10-29 22:42:50.296263291 +0000 UTC m=+1709.207706848"
Oct 29 22:42:50.373538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:50.373504    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:50.374606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:50.374581    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:42:53.928145 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:53.928103    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"]
Oct 29 22:42:53.933084 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:53.933059    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"
Oct 29 22:42:53.954020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:53.953993    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2a3ef0\""
Oct 29 22:42:53.976280 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:53.976242    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"]
Oct 29 22:42:53.976517 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:53.976497    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2a3ef0-dockercfg-w4lzq\""
Oct 29 22:42:53.988465 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:53.988436    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"]
Oct 29 22:42:53.993236 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:53.993211    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"
Oct 29 22:42:54.022163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:54.018660    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4109193-5cc4-4dcb-9f8d-fbbe5e359010-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v\" (UID: \"a4109193-5cc4-4dcb-9f8d-fbbe5e359010\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"
Oct 29 22:42:54.022163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:54.018930    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz9kg\" (UniqueName: \"kubernetes.io/projected/a4109193-5cc4-4dcb-9f8d-fbbe5e359010-kube-api-access-wz9kg\") pod \"payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v\" (UID: \"a4109193-5cc4-4dcb-9f8d-fbbe5e359010\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"
Oct 29 22:42:54.024780 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:54.024755    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"]
Oct 29 22:42:54.214891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:54.169911    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz9kg\" (UniqueName: \"kubernetes.io/projected/a4109193-5cc4-4dcb-9f8d-fbbe5e359010-kube-api-access-wz9kg\") pod \"payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v\" (UID: \"a4109193-5cc4-4dcb-9f8d-fbbe5e359010\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"
Oct 29 22:42:54.214891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:54.170037    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b19e1fbe-b078-4a6b-a2c0-c30f09df2962-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv\" (UID: \"b19e1fbe-b078-4a6b-a2c0-c30f09df2962\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"
Oct 29 22:42:54.214891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:54.170104    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnl7n\" (UniqueName: \"kubernetes.io/projected/b19e1fbe-b078-4a6b-a2c0-c30f09df2962-kube-api-access-hnl7n\") pod \"payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv\" (UID: \"b19e1fbe-b078-4a6b-a2c0-c30f09df2962\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"
Oct 29 22:42:54.214891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:54.170181    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4109193-5cc4-4dcb-9f8d-fbbe5e359010-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v\" (UID: \"a4109193-5cc4-4dcb-9f8d-fbbe5e359010\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"
Oct 29 22:42:54.214891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:54.171247    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4109193-5cc4-4dcb-9f8d-fbbe5e359010-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v\" (UID: \"a4109193-5cc4-4dcb-9f8d-fbbe5e359010\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"
Oct 29 22:42:54.287788 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:54.285894    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b19e1fbe-b078-4a6b-a2c0-c30f09df2962-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv\" (UID: \"b19e1fbe-b078-4a6b-a2c0-c30f09df2962\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"
Oct 29 22:42:54.287788 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:54.285985    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnl7n\" (UniqueName: \"kubernetes.io/projected/b19e1fbe-b078-4a6b-a2c0-c30f09df2962-kube-api-access-hnl7n\") pod \"payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv\" (UID: \"b19e1fbe-b078-4a6b-a2c0-c30f09df2962\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"
Oct 29 22:42:54.287788 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:54.287624    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b19e1fbe-b078-4a6b-a2c0-c30f09df2962-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv\" (UID: \"b19e1fbe-b078-4a6b-a2c0-c30f09df2962\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"
Oct 29 22:42:54.379572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:54.376699    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnl7n\" (UniqueName: \"kubernetes.io/projected/b19e1fbe-b078-4a6b-a2c0-c30f09df2962-kube-api-access-hnl7n\") pod \"payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv\" (UID: \"b19e1fbe-b078-4a6b-a2c0-c30f09df2962\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"
Oct 29 22:42:54.389617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:54.389536    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz9kg\" (UniqueName: \"kubernetes.io/projected/a4109193-5cc4-4dcb-9f8d-fbbe5e359010-kube-api-access-wz9kg\") pod \"payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v\" (UID: \"a4109193-5cc4-4dcb-9f8d-fbbe5e359010\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"
Oct 29 22:42:54.552386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:54.552292    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"
Oct 29 22:42:54.613833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:54.613698    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"
Oct 29 22:42:54.880929 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:42:54.880885    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4109193_5cc4_4dcb_9f8d_fbbe5e359010.slice/crio-9cb99e3fc23dcda23c473a1a81fb306078ff88ab5fe5cc07ff5a2ff7043b40a8 WatchSource:0}: Error finding container 9cb99e3fc23dcda23c473a1a81fb306078ff88ab5fe5cc07ff5a2ff7043b40a8: Status 404 returned error can't find the container with id 9cb99e3fc23dcda23c473a1a81fb306078ff88ab5fe5cc07ff5a2ff7043b40a8
Oct 29 22:42:54.882771 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:54.882744    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"]
Oct 29 22:42:54.947165 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:54.947128    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"]
Oct 29 22:42:54.949327 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:42:54.949290    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19e1fbe_b078_4a6b_a2c0_c30f09df2962.slice/crio-2137b71526f8ee013a579d9c327db99162d7688bcedf97579a61eed82492a772 WatchSource:0}: Error finding container 2137b71526f8ee013a579d9c327db99162d7688bcedf97579a61eed82492a772: Status 404 returned error can't find the container with id 2137b71526f8ee013a579d9c327db99162d7688bcedf97579a61eed82492a772
Oct 29 22:42:55.473888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:55.473840    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh" event={"ID":"b6cffbad-4429-4e61-b582-5b45fda07b48","Type":"ContainerDied","Data":"11457e1ec66e14af1203fc2fe0d0bd584389b64c10606d5266cef0f8a5f31ffb"}
Oct 29 22:42:55.474104 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:55.473802    4559 generic.go:356] "Generic (PLEG): container finished" podID="b6cffbad-4429-4e61-b582-5b45fda07b48" containerID="11457e1ec66e14af1203fc2fe0d0bd584389b64c10606d5266cef0f8a5f31ffb" exitCode=0
Oct 29 22:42:55.474440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:55.474421    4559 scope.go:122] "RemoveContainer" containerID="11457e1ec66e14af1203fc2fe0d0bd584389b64c10606d5266cef0f8a5f31ffb"
Oct 29 22:42:55.476609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:55.476582    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v" event={"ID":"a4109193-5cc4-4dcb-9f8d-fbbe5e359010","Type":"ContainerStarted","Data":"d69054a2db02d913431abd772f1ed9f865efd4bc1a2b8659adb8f86ec6625d06"}
Oct 29 22:42:55.476731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:55.476627    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v" event={"ID":"a4109193-5cc4-4dcb-9f8d-fbbe5e359010","Type":"ContainerStarted","Data":"9cb99e3fc23dcda23c473a1a81fb306078ff88ab5fe5cc07ff5a2ff7043b40a8"}
Oct 29 22:42:55.476731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:55.476654    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"
Oct 29 22:42:55.476928 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:55.476893    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:42:55.477006 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:55.476987    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:42:55.477065 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:55.477039    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:42:55.478621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:55.478601    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv" event={"ID":"b19e1fbe-b078-4a6b-a2c0-c30f09df2962","Type":"ContainerStarted","Data":"b37f46a6ab9ce78d5a80188de18ac7b3a3755b3b9f0832800c8b5945a4a7b8fd"}
Oct 29 22:42:55.478797 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:55.478773    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv" event={"ID":"b19e1fbe-b078-4a6b-a2c0-c30f09df2962","Type":"ContainerStarted","Data":"2137b71526f8ee013a579d9c327db99162d7688bcedf97579a61eed82492a772"}
Oct 29 22:42:55.478953 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:55.478854    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"
Oct 29 22:42:55.579859 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:55.579782    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v" podStartSLOduration=2.579763001 podStartE2EDuration="2.579763001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:55.57854806 +0000 UTC m=+1714.489991617" watchObservedRunningTime="2026-10-29 22:42:55.579763001 +0000 UTC m=+1714.491206557"
Oct 29 22:42:55.622682 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:55.622620    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv" podStartSLOduration=2.6226015179999997 podStartE2EDuration="2.622601518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:55.621167408 +0000 UTC m=+1714.532610964" watchObservedRunningTime="2026-10-29 22:42:55.622601518 +0000 UTC m=+1714.534045073"
Oct 29 22:42:56.490519 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.490479    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh" event={"ID":"b6cffbad-4429-4e61-b582-5b45fda07b48","Type":"ContainerStarted","Data":"1d4f576a2dabe48747affe92fdca5d08de1f6ecbba49e7fdce2e3da1d56690c0"}
Oct 29 22:42:56.689939 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.689902    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"]
Oct 29 22:42:56.694796 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.694767    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.748206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.748093    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c7d192-openshift-default-dockercfg-v9rsq\""
Oct 29 22:42:56.764231 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.764181    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.765271 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.765243    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.765516 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.765500    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.765644 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.765629    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.778333 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.778299    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.778684 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.778664    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-278wj\" (UniqueName: \"kubernetes.io/projected/6bc838bd-671f-463d-b98e-0435f56ff70e-kube-api-access-278wj\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.779002 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.778979    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc838bd-671f-463d-b98e-0435f56ff70e-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.779212 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.779195    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.864305 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.864272    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.889920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.889878    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"]
Oct 29 22:42:56.965823 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.965766    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc838bd-671f-463d-b98e-0435f56ff70e-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.966037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.965837    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.966037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.965900    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.966037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.965996    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.966037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.966023    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.966264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.966061    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.966264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.966085    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.966264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.966101    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.966264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.966130    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-278wj\" (UniqueName: \"kubernetes.io/projected/6bc838bd-671f-463d-b98e-0435f56ff70e-kube-api-access-278wj\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.967236 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.966802    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.967236 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.966834    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.967236 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.967143    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.967477 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.967419    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.967723 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.967687    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc838bd-671f-463d-b98e-0435f56ff70e-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.970163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.970129    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.970292 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.970204    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.991864 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.991831    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-278wj\" (UniqueName: \"kubernetes.io/projected/6bc838bd-671f-463d-b98e-0435f56ff70e-kube-api-access-278wj\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:56.992767 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:56.992749    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:57.012197 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:57.012128    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:57.390607 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:57.390578    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"]
Oct 29 22:42:57.391896 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:42:57.391864    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc838bd_671f_463d_b98e_0435f56ff70e.slice/crio-1e85493a94f69622ef354206c6c088e2ec5b865683c8549dbf217d2bec8bad34 WatchSource:0}: Error finding container 1e85493a94f69622ef354206c6c088e2ec5b865683c8549dbf217d2bec8bad34: Status 404 returned error can't find the container with id 1e85493a94f69622ef354206c6c088e2ec5b865683c8549dbf217d2bec8bad34
Oct 29 22:42:57.394803 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:57.394770    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:42:57.394958 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:57.394867    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:42:57.394958 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:57.394899    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:42:57.397487 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:57.397460    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:42:57.398765 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:57.398740    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 29 22:42:57.398886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:57.398784    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh" podUID="b6cffbad-4429-4e61-b582-5b45fda07b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 29 22:42:57.504672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:57.504622    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m" event={"ID":"6bc838bd-671f-463d-b98e-0435f56ff70e","Type":"ContainerStarted","Data":"1e85493a94f69622ef354206c6c088e2ec5b865683c8549dbf217d2bec8bad34"}
Oct 29 22:42:57.722893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:57.722853    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:42:58.407393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:58.407351    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 29 22:42:58.407886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:58.407855    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh" podUID="b6cffbad-4429-4e61-b582-5b45fda07b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 29 22:42:59.001837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:58.995939    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m" event={"ID":"6bc838bd-671f-463d-b98e-0435f56ff70e","Type":"ContainerStarted","Data":"c8a7ca9b85679ed0ffdda77d56a96a572fc3af922d07a5da6f66a93014d16594"}
Oct 29 22:42:59.012529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:59.012484    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:42:59.014503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:59.014472    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 29 22:42:59.014648 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:59.014526    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m" podUID="6bc838bd-671f-463d-b98e-0435f56ff70e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 29 22:42:59.087840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:59.086871    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m" podStartSLOduration=3.08684851 podStartE2EDuration="3.08684851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:59.081729262 +0000 UTC m=+1717.993172841" watchObservedRunningTime="2026-10-29 22:42:59.08684851 +0000 UTC m=+1717.998292077"
Oct 29 22:42:59.401050 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:59.401014    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 29 22:42:59.401232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:42:59.401073    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh" podUID="b6cffbad-4429-4e61-b582-5b45fda07b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 29 22:43:00.013483 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:00.013446    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 29 22:43:00.013914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:00.013503    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m" podUID="6bc838bd-671f-463d-b98e-0435f56ff70e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 29 22:43:00.424621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:00.424587    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 29 22:43:00.425180 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:00.425139    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh" podUID="b6cffbad-4429-4e61-b582-5b45fda07b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 29 22:43:01.012549 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:01.012505    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 29 22:43:01.012762 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:01.012581    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m" podUID="6bc838bd-671f-463d-b98e-0435f56ff70e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 29 22:43:01.399054 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:01.398572    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 29 22:43:01.399054 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:01.398644    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh" podUID="b6cffbad-4429-4e61-b582-5b45fda07b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 29 22:43:02.040834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:02.038114    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 29 22:43:02.040834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:02.038200    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m" podUID="6bc838bd-671f-463d-b98e-0435f56ff70e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 29 22:43:02.397977 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:02.397935    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 29 22:43:02.402082 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:02.402039    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh" podUID="b6cffbad-4429-4e61-b582-5b45fda07b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 29 22:43:02.809894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:02.805234    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:03.042545 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:03.037163    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 29 22:43:03.042545 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:03.037263    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m" podUID="6bc838bd-671f-463d-b98e-0435f56ff70e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 29 22:43:03.786069 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:03.786028    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:43:03.788993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:03.788962    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:43:04.013669 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:04.013629    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 29 22:43:04.013872 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:04.013699    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m" podUID="6bc838bd-671f-463d-b98e-0435f56ff70e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 29 22:43:04.990024 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:04.989983    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"]
Oct 29 22:43:04.997631 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:04.997602    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.012614 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.012588    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 29 22:43:05.012830 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.012789    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m" podUID="6bc838bd-671f-463d-b98e-0435f56ff70e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 29 22:43:05.021037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.021012    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-6fc6bd-openshift-default-dockercfg-mff5d\""
Oct 29 22:43:05.071116 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.071084    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"]
Oct 29 22:43:05.117377 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.117322    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-envoy\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.117591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.117447    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-token\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.117591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.117489    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-workload-certs\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.117591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.117575    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istiod-ca-cert\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.117788 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.117637    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-podinfo\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.117788 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.117670    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-data\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.117944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.117854    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-credential-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.117944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.117931    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-workload-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.118053 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.118040    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kppnp\" (UniqueName: \"kubernetes.io/projected/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-kube-api-access-kppnp\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.229072 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.229027    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-envoy\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.251875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.251772    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-token\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.252440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.252415    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-workload-certs\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.267554 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.267523    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istiod-ca-cert\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.268591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.268566    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-podinfo\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.268801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.268780    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-data\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.269052 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.269033    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-credential-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.269226 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.269212    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-workload-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.269476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.269441    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kppnp\" (UniqueName: \"kubernetes.io/projected/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-kube-api-access-kppnp\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.304837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.281752    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-data\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.304837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.257151    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-envoy\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.304837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.268483    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istiod-ca-cert\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.304837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.252889    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-workload-certs\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.304837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.284368    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-workload-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.304837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.284618    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-credential-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.531885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.500744    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kppnp\" (UniqueName: \"kubernetes.io/projected/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-kube-api-access-kppnp\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.736457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.721719    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-token\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.736457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.723721    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-podinfo\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:05.919124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:05.919088    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:06.026913 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:06.023145    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:43:06.026913 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:06.025597    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:43:06.026913 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:06.026789    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:43:06.273184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:06.273148    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"]
Oct 29 22:43:06.274356 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:43:06.274324    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3385fd_f8ce_46b7_8ea2_cffddd2cd9a6.slice/crio-39f35ed5b5f7f7d85abcb0dd9143c07d0d2a978f32f1869fdc71b8824c58cb43 WatchSource:0}: Error finding container 39f35ed5b5f7f7d85abcb0dd9143c07d0d2a978f32f1869fdc71b8824c58cb43: Status 404 returned error can't find the container with id 39f35ed5b5f7f7d85abcb0dd9143c07d0d2a978f32f1869fdc71b8824c58cb43
Oct 29 22:43:06.278493 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:06.278450    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:43:06.278614 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:06.278547    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:43:06.278614 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:06.278591    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:43:06.504963 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:06.504926    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"
Oct 29 22:43:06.507994 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:06.507963    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"
Oct 29 22:43:07.183726 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:07.183686    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v" event={"ID":"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6","Type":"ContainerStarted","Data":"234e547bc5821a880a916bd6b684de6dbee12b5377dffafb082c7b20aaf3ea48"}
Oct 29 22:43:07.184194 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:07.183752    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v" event={"ID":"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6","Type":"ContainerStarted","Data":"39f35ed5b5f7f7d85abcb0dd9143c07d0d2a978f32f1869fdc71b8824c58cb43"}
Oct 29 22:43:07.234345 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:07.234291    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v" podStartSLOduration=3.234278399 podStartE2EDuration="3.234278399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:07.231579992 +0000 UTC m=+1726.143023545" watchObservedRunningTime="2026-10-29 22:43:07.234278399 +0000 UTC m=+1726.145722014"
Oct 29 22:43:07.920028 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:07.919992    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:07.926367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:07.926337    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:08.006082 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:08.006038    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"]
Oct 29 22:43:08.006297 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:08.006265    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerName="istio-proxy" containerID="cri-o://eecdbee41c5e6d5266d0f31e3c7e43f1769441df8f73e406f3d902d5b8c17f21" gracePeriod=30
Oct 29 22:43:08.193155 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:08.193066    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:08.194009 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:08.193985    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:43:11.355343 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:11.355296    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"]
Oct 29 22:43:11.362023 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:11.361993    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"
Oct 29 22:43:11.382212 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:11.382180    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"]
Oct 29 22:43:11.393000 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:11.392965    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-05417d-dockercfg-p8g42\""
Oct 29 22:43:11.478856 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:11.478795    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-05417d\""
Oct 29 22:43:11.613819 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:11.583661    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwdhf\" (UniqueName: \"kubernetes.io/projected/02cf8b24-105b-4675-a4de-8c8c77e4afa9-kube-api-access-rwdhf\") pod \"payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4\" (UID: \"02cf8b24-105b-4675-a4de-8c8c77e4afa9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"
Oct 29 22:43:11.613819 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:11.583827    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cf8b24-105b-4675-a4de-8c8c77e4afa9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4\" (UID: \"02cf8b24-105b-4675-a4de-8c8c77e4afa9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"
Oct 29 22:43:11.747204 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:11.747145    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwdhf\" (UniqueName: \"kubernetes.io/projected/02cf8b24-105b-4675-a4de-8c8c77e4afa9-kube-api-access-rwdhf\") pod \"payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4\" (UID: \"02cf8b24-105b-4675-a4de-8c8c77e4afa9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"
Oct 29 22:43:11.802938 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:11.802902    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cf8b24-105b-4675-a4de-8c8c77e4afa9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4\" (UID: \"02cf8b24-105b-4675-a4de-8c8c77e4afa9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"
Oct 29 22:43:11.806556 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:11.806486    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cf8b24-105b-4675-a4de-8c8c77e4afa9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4\" (UID: \"02cf8b24-105b-4675-a4de-8c8c77e4afa9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"
Oct 29 22:43:11.842912 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:11.839877    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwdhf\" (UniqueName: \"kubernetes.io/projected/02cf8b24-105b-4675-a4de-8c8c77e4afa9-kube-api-access-rwdhf\") pod \"payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4\" (UID: \"02cf8b24-105b-4675-a4de-8c8c77e4afa9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"
Oct 29 22:43:11.982951 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:11.982911    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"
Oct 29 22:43:12.365079 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:12.365053    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"]
Oct 29 22:43:13.258735 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.256503    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4" event={"ID":"02cf8b24-105b-4675-a4de-8c8c77e4afa9","Type":"ContainerStarted","Data":"31959f840e48a0b205d094e363cd6d9c6c44aadc8c280711d0f2e02b86157326"}
Oct 29 22:43:13.258735 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.256572    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4" event={"ID":"02cf8b24-105b-4675-a4de-8c8c77e4afa9","Type":"ContainerStarted","Data":"d2871d8061d02d663158acc201ba5f6d4830a292f71ab3e68ab76e1892dafc2b"}
Oct 29 22:43:13.258735 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.256631    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"
Oct 29 22:43:13.259929 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.259899    4559 generic.go:356] "Generic (PLEG): container finished" podID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerID="eecdbee41c5e6d5266d0f31e3c7e43f1769441df8f73e406f3d902d5b8c17f21" exitCode=0
Oct 29 22:43:13.260350 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.260332    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" event={"ID":"1b88f5e2-5192-4da4-8f7c-51db3c907706","Type":"ContainerDied","Data":"eecdbee41c5e6d5266d0f31e3c7e43f1769441df8f73e406f3d902d5b8c17f21"}
Oct 29 22:43:13.260541 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.260520    4559 scope.go:122] "RemoveContainer" containerID="8a38048bad7695b501921c2f9341d014c3e649c9add4c1963958ab9812e34ae1"
Oct 29 22:43:13.326292 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.326245    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4" podStartSLOduration=2.326231636 podStartE2EDuration="2.326231636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:13.319310718 +0000 UTC m=+1732.230754274" watchObservedRunningTime="2026-10-29 22:43:13.326231636 +0000 UTC m=+1732.237675190"
Oct 29 22:43:13.339638 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.339609    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:43:13.478647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.478605    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-data\") pod \"1b88f5e2-5192-4da4-8f7c-51db3c907706\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") "
Oct 29 22:43:13.479200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.478704    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-credential-socket\") pod \"1b88f5e2-5192-4da4-8f7c-51db3c907706\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") "
Oct 29 22:43:13.479200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.478747    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b88f5e2-5192-4da4-8f7c-51db3c907706-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b88f5e2-5192-4da4-8f7c-51db3c907706-istiod-ca-cert\") pod \"1b88f5e2-5192-4da4-8f7c-51db3c907706\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") "
Oct 29 22:43:13.479200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.478858    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-workload-socket\") pod \"1b88f5e2-5192-4da4-8f7c-51db3c907706\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") "
Oct 29 22:43:13.479200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.478970    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-data" pod "1b88f5e2-5192-4da4-8f7c-51db3c907706" (UID: "1b88f5e2-5192-4da4-8f7c-51db3c907706"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:13.479200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.479044    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-credential-socket" pod "1b88f5e2-5192-4da4-8f7c-51db3c907706" (UID: "1b88f5e2-5192-4da4-8f7c-51db3c907706"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:13.479200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.479096    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-podinfo\") pod \"1b88f5e2-5192-4da4-8f7c-51db3c907706\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") "
Oct 29 22:43:13.479200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.479149    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b88f5e2-5192-4da4-8f7c-51db3c907706-kube-api-access-h8rrd\" (UniqueName: \"kubernetes.io/projected/1b88f5e2-5192-4da4-8f7c-51db3c907706-kube-api-access-h8rrd\") pod \"1b88f5e2-5192-4da4-8f7c-51db3c907706\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") "
Oct 29 22:43:13.479612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.479229    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-token\" (UniqueName: \"kubernetes.io/projected/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-token\") pod \"1b88f5e2-5192-4da4-8f7c-51db3c907706\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") "
Oct 29 22:43:13.479612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.479300    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-workload-certs\") pod \"1b88f5e2-5192-4da4-8f7c-51db3c907706\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") "
Oct 29 22:43:13.479612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.479426    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-envoy\") pod \"1b88f5e2-5192-4da4-8f7c-51db3c907706\" (UID: \"1b88f5e2-5192-4da4-8f7c-51db3c907706\") "
Oct 29 22:43:13.479795 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.479298    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-workload-socket" pod "1b88f5e2-5192-4da4-8f7c-51db3c907706" (UID: "1b88f5e2-5192-4da4-8f7c-51db3c907706"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:13.479795 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.479351    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b88f5e2-5192-4da4-8f7c-51db3c907706-istiod-ca-cert" pod "1b88f5e2-5192-4da4-8f7c-51db3c907706" (UID: "1b88f5e2-5192-4da4-8f7c-51db3c907706"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:43:13.479931 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.479846    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-workload-certs" pod "1b88f5e2-5192-4da4-8f7c-51db3c907706" (UID: "1b88f5e2-5192-4da4-8f7c-51db3c907706"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:13.480298 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.480271    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.480582 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.480303    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.480582 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.480317    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.480582 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.480331    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b88f5e2-5192-4da4-8f7c-51db3c907706-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.480582 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.480344    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.483439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.482803    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-envoy" pod "1b88f5e2-5192-4da4-8f7c-51db3c907706" (UID: "1b88f5e2-5192-4da4-8f7c-51db3c907706"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:13.483439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.483106    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b88f5e2-5192-4da4-8f7c-51db3c907706-kube-api-access-h8rrd" pod "1b88f5e2-5192-4da4-8f7c-51db3c907706" (UID: "1b88f5e2-5192-4da4-8f7c-51db3c907706"). InnerVolumeSpecName "kube-api-access-h8rrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:43:13.483439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.483179    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-podinfo" pod "1b88f5e2-5192-4da4-8f7c-51db3c907706" (UID: "1b88f5e2-5192-4da4-8f7c-51db3c907706"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:43:13.483686 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.483519    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-token" pod "1b88f5e2-5192-4da4-8f7c-51db3c907706" (UID: "1b88f5e2-5192-4da4-8f7c-51db3c907706"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:43:13.581621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.581533    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.581621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.581566    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.581621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.581578    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b88f5e2-5192-4da4-8f7c-51db3c907706-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.581621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:13.581590    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8rrd\" (UniqueName: \"kubernetes.io/projected/1b88f5e2-5192-4da4-8f7c-51db3c907706-kube-api-access-h8rrd\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:13.834889 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:43:13.832850    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:43:14.269961 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.269925    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"
Oct 29 22:43:14.270133 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.269954    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc" event={"ID":"1b88f5e2-5192-4da4-8f7c-51db3c907706","Type":"ContainerDied","Data":"4225214f87307f078aff41307650488f02a343ce01d9992dfb828315b6884f60"}
Oct 29 22:43:14.270133 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.270016    4559 scope.go:122] "RemoveContainer" containerID="eecdbee41c5e6d5266d0f31e3c7e43f1769441df8f73e406f3d902d5b8c17f21"
Oct 29 22:43:14.345079 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.345043    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"]
Oct 29 22:43:14.356516 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.356473    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-5lkhc"]
Oct 29 22:43:14.549764 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.549668    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"]
Oct 29 22:43:14.566193 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.566153    4559 cpu_manager.go:414] "containerMap: removing container" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerName="istio-proxy"
Oct 29 22:43:14.582895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.582855    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerName="istio-proxy"
Oct 29 22:43:14.583500 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.583474    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerName="istio-proxy"
Oct 29 22:43:14.706836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.700026    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"]
Oct 29 22:43:14.706836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.701448    4559 cpu_manager.go:414] "containerMap: removing container" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerName="istio-proxy"
Oct 29 22:43:14.706836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.701473    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerName="istio-proxy"
Oct 29 22:43:14.706836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.701652    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" containerName="istio-proxy"
Oct 29 22:43:14.772841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.744001    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"
Oct 29 22:43:14.838892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.835542    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-6fc6bd-dockercfg-rl2hk\""
Oct 29 22:43:14.874681 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.861364    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8vc9\" (UniqueName: \"kubernetes.io/projected/7728dea4-21ee-478e-8fcc-c4632e1b350a-kube-api-access-x8vc9\") pod \"payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh\" (UID: \"7728dea4-21ee-478e-8fcc-c4632e1b350a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"
Oct 29 22:43:14.878055 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.877341    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7728dea4-21ee-478e-8fcc-c4632e1b350a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh\" (UID: \"7728dea4-21ee-478e-8fcc-c4632e1b350a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"
Oct 29 22:43:14.878238 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.870635    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-6fc6bd\""
Oct 29 22:43:14.938484 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.938289    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"]
Oct 29 22:43:14.938484 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.938345    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"]
Oct 29 22:43:14.938484 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.938387    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"
Oct 29 22:43:14.979331 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.979289    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8vc9\" (UniqueName: \"kubernetes.io/projected/7728dea4-21ee-478e-8fcc-c4632e1b350a-kube-api-access-x8vc9\") pod \"payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh\" (UID: \"7728dea4-21ee-478e-8fcc-c4632e1b350a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"
Oct 29 22:43:14.979527 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.979416    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7728dea4-21ee-478e-8fcc-c4632e1b350a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh\" (UID: \"7728dea4-21ee-478e-8fcc-c4632e1b350a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"
Oct 29 22:43:14.980398 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:14.980370    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7728dea4-21ee-478e-8fcc-c4632e1b350a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh\" (UID: \"7728dea4-21ee-478e-8fcc-c4632e1b350a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"
Oct 29 22:43:15.016772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:15.016738    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8vc9\" (UniqueName: \"kubernetes.io/projected/7728dea4-21ee-478e-8fcc-c4632e1b350a-kube-api-access-x8vc9\") pod \"payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh\" (UID: \"7728dea4-21ee-478e-8fcc-c4632e1b350a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"
Oct 29 22:43:15.081453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:15.081405    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n\" (UID: \"2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"
Oct 29 22:43:15.081913 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:15.081876    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r6cx\" (UniqueName: \"kubernetes.io/projected/2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0-kube-api-access-6r6cx\") pod \"payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n\" (UID: \"2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"
Oct 29 22:43:15.183571 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:15.183527    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r6cx\" (UniqueName: \"kubernetes.io/projected/2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0-kube-api-access-6r6cx\") pod \"payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n\" (UID: \"2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"
Oct 29 22:43:15.183789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:15.183680    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n\" (UID: \"2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"
Oct 29 22:43:15.184395 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:15.184368    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n\" (UID: \"2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"
Oct 29 22:43:15.239919 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:15.239880    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b88f5e2-5192-4da4-8f7c-51db3c907706" path="/var/lib/kubelet/pods/1b88f5e2-5192-4da4-8f7c-51db3c907706/volumes"
Oct 29 22:43:15.240113 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:15.240090    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"
Oct 29 22:43:15.244623 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:15.244529    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r6cx\" (UniqueName: \"kubernetes.io/projected/2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0-kube-api-access-6r6cx\") pod \"payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n\" (UID: \"2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"
Oct 29 22:43:15.255548 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:15.255521    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"
Oct 29 22:43:15.577491 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:15.577458    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"]
Oct 29 22:43:15.582245 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:43:15.582204    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7728dea4_21ee_478e_8fcc_c4632e1b350a.slice/crio-6be1ddbef22c67c18ff959397bed357a682285da004c6cf73806b2740b3620f7 WatchSource:0}: Error finding container 6be1ddbef22c67c18ff959397bed357a682285da004c6cf73806b2740b3620f7: Status 404 returned error can't find the container with id 6be1ddbef22c67c18ff959397bed357a682285da004c6cf73806b2740b3620f7
Oct 29 22:43:15.604865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:15.604835    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"]
Oct 29 22:43:15.606509 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:43:15.606477    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4bcb98_02b3_4b0d_9f4f_6b5e6976fdc0.slice/crio-b831ce0e1df06cc6ca900702b1d36521a225162f4842e2db0fea959e4da51e2d WatchSource:0}: Error finding container b831ce0e1df06cc6ca900702b1d36521a225162f4842e2db0fea959e4da51e2d: Status 404 returned error can't find the container with id b831ce0e1df06cc6ca900702b1d36521a225162f4842e2db0fea959e4da51e2d
Oct 29 22:43:16.295404 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:16.295369    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n" event={"ID":"2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0","Type":"ContainerStarted","Data":"8ffc843b4979e16d3734b097d84e9798f3d086242c47d4024e5b390806845ff3"}
Oct 29 22:43:16.295645 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:16.295430    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n" event={"ID":"2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0","Type":"ContainerStarted","Data":"b831ce0e1df06cc6ca900702b1d36521a225162f4842e2db0fea959e4da51e2d"}
Oct 29 22:43:16.295645 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:16.295460    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"
Oct 29 22:43:16.297557 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:16.297531    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh" event={"ID":"7728dea4-21ee-478e-8fcc-c4632e1b350a","Type":"ContainerStarted","Data":"25e990b39d274eaf0eedb28be6d181b7ba8043740fa224f2555de6b4c93207e8"}
Oct 29 22:43:16.297702 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:16.297576    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh" event={"ID":"7728dea4-21ee-478e-8fcc-c4632e1b350a","Type":"ContainerStarted","Data":"6be1ddbef22c67c18ff959397bed357a682285da004c6cf73806b2740b3620f7"}
Oct 29 22:43:16.297702 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:16.297631    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"
Oct 29 22:43:16.346692 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:16.346634    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n" podStartSLOduration=2.346619176 podStartE2EDuration="2.346619176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:16.34291804 +0000 UTC m=+1735.254361593" watchObservedRunningTime="2026-10-29 22:43:16.346619176 +0000 UTC m=+1735.258062729"
Oct 29 22:43:16.391854 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:16.391784    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh" podStartSLOduration=2.39176852 podStartE2EDuration="2.39176852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:16.388744455 +0000 UTC m=+1735.300188079" watchObservedRunningTime="2026-10-29 22:43:16.39176852 +0000 UTC m=+1735.303212074"
Oct 29 22:43:22.155047 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:22.155007    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"]
Oct 29 22:43:22.155655 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:22.155278    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl" podUID="220a1952-3291-485e-b6d4-4336b1cd1db3" containerName="istio-proxy" containerID="cri-o://ce451b6c74fb7c2ccf5cb17f78c6a3a2f5800da4992e7b1ccdd2c0b9c1abe1eb" gracePeriod=30
Oct 29 22:43:24.274179 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:24.274145    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"
Oct 29 22:43:27.333954 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.333922    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"
Oct 29 22:43:27.335284 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.335249    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"
Oct 29 22:43:27.493883 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.493802    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:43:27.575499 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.575459    4559 generic.go:356] "Generic (PLEG): container finished" podID="220a1952-3291-485e-b6d4-4336b1cd1db3" containerID="ce451b6c74fb7c2ccf5cb17f78c6a3a2f5800da4992e7b1ccdd2c0b9c1abe1eb" exitCode=0
Oct 29 22:43:27.575740 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.575591    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"
Oct 29 22:43:27.575740 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.575590    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl" event={"ID":"220a1952-3291-485e-b6d4-4336b1cd1db3","Type":"ContainerDied","Data":"ce451b6c74fb7c2ccf5cb17f78c6a3a2f5800da4992e7b1ccdd2c0b9c1abe1eb"}
Oct 29 22:43:27.575740 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.575729    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl" event={"ID":"220a1952-3291-485e-b6d4-4336b1cd1db3","Type":"ContainerDied","Data":"8a29a65d1a23a3450e66fa59a40ed24f33ff2a3ceb23f5d450a919a4a47d5a4b"}
Oct 29 22:43:27.576017 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.575764    4559 scope.go:122] "RemoveContainer" containerID="ce451b6c74fb7c2ccf5cb17f78c6a3a2f5800da4992e7b1ccdd2c0b9c1abe1eb"
Oct 29 22:43:27.587383 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.587347    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-data\") pod \"220a1952-3291-485e-b6d4-4336b1cd1db3\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") "
Oct 29 22:43:27.587719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.587442    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-podinfo\") pod \"220a1952-3291-485e-b6d4-4336b1cd1db3\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") "
Oct 29 22:43:27.587719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.587496    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-credential-socket\") pod \"220a1952-3291-485e-b6d4-4336b1cd1db3\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") "
Oct 29 22:43:27.587719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.587618    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-workload-socket\") pod \"220a1952-3291-485e-b6d4-4336b1cd1db3\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") "
Oct 29 22:43:27.587719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.587660    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-token\" (UniqueName: \"kubernetes.io/projected/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-token\") pod \"220a1952-3291-485e-b6d4-4336b1cd1db3\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") "
Oct 29 22:43:27.587719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.587700    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220a1952-3291-485e-b6d4-4336b1cd1db3-kube-api-access-v94tv\" (UniqueName: \"kubernetes.io/projected/220a1952-3291-485e-b6d4-4336b1cd1db3-kube-api-access-v94tv\") pod \"220a1952-3291-485e-b6d4-4336b1cd1db3\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") "
Oct 29 22:43:27.588102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.587784    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-data" pod "220a1952-3291-485e-b6d4-4336b1cd1db3" (UID: "220a1952-3291-485e-b6d4-4336b1cd1db3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:27.588102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.588026    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-envoy\") pod \"220a1952-3291-485e-b6d4-4336b1cd1db3\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") "
Oct 29 22:43:27.588102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.588073    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/220a1952-3291-485e-b6d4-4336b1cd1db3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220a1952-3291-485e-b6d4-4336b1cd1db3-istiod-ca-cert\") pod \"220a1952-3291-485e-b6d4-4336b1cd1db3\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") "
Oct 29 22:43:27.588261 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.588147    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-workload-certs\") pod \"220a1952-3291-485e-b6d4-4336b1cd1db3\" (UID: \"220a1952-3291-485e-b6d4-4336b1cd1db3\") "
Oct 29 22:43:27.589394 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.588984    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:27.589394 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.588026    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-credential-socket" pod "220a1952-3291-485e-b6d4-4336b1cd1db3" (UID: "220a1952-3291-485e-b6d4-4336b1cd1db3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:27.589394 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.588726    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-workload-socket" pod "220a1952-3291-485e-b6d4-4336b1cd1db3" (UID: "220a1952-3291-485e-b6d4-4336b1cd1db3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:27.589394 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.589187    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-workload-certs" pod "220a1952-3291-485e-b6d4-4336b1cd1db3" (UID: "220a1952-3291-485e-b6d4-4336b1cd1db3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:27.589394 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.589365    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220a1952-3291-485e-b6d4-4336b1cd1db3-istiod-ca-cert" pod "220a1952-3291-485e-b6d4-4336b1cd1db3" (UID: "220a1952-3291-485e-b6d4-4336b1cd1db3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:43:27.590710 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.590680    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-podinfo" pod "220a1952-3291-485e-b6d4-4336b1cd1db3" (UID: "220a1952-3291-485e-b6d4-4336b1cd1db3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:43:27.590932 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.590910    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-token" pod "220a1952-3291-485e-b6d4-4336b1cd1db3" (UID: "220a1952-3291-485e-b6d4-4336b1cd1db3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:43:27.592803 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.592750    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220a1952-3291-485e-b6d4-4336b1cd1db3-kube-api-access-v94tv" pod "220a1952-3291-485e-b6d4-4336b1cd1db3" (UID: "220a1952-3291-485e-b6d4-4336b1cd1db3"). InnerVolumeSpecName "kube-api-access-v94tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:43:27.592803 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.592778    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-envoy" pod "220a1952-3291-485e-b6d4-4336b1cd1db3" (UID: "220a1952-3291-485e-b6d4-4336b1cd1db3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:27.633408 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.633387    4559 scope.go:122] "RemoveContainer" containerID="ce451b6c74fb7c2ccf5cb17f78c6a3a2f5800da4992e7b1ccdd2c0b9c1abe1eb"
Oct 29 22:43:27.633785 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:43:27.633755    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce451b6c74fb7c2ccf5cb17f78c6a3a2f5800da4992e7b1ccdd2c0b9c1abe1eb\": container with ID starting with ce451b6c74fb7c2ccf5cb17f78c6a3a2f5800da4992e7b1ccdd2c0b9c1abe1eb not found: ID does not exist" containerID="ce451b6c74fb7c2ccf5cb17f78c6a3a2f5800da4992e7b1ccdd2c0b9c1abe1eb"
Oct 29 22:43:27.633917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.633795    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce451b6c74fb7c2ccf5cb17f78c6a3a2f5800da4992e7b1ccdd2c0b9c1abe1eb"} err="failed to get container status \"ce451b6c74fb7c2ccf5cb17f78c6a3a2f5800da4992e7b1ccdd2c0b9c1abe1eb\": rpc error: code = NotFound desc = could not find container \"ce451b6c74fb7c2ccf5cb17f78c6a3a2f5800da4992e7b1ccdd2c0b9c1abe1eb\": container with ID starting with ce451b6c74fb7c2ccf5cb17f78c6a3a2f5800da4992e7b1ccdd2c0b9c1abe1eb not found: ID does not exist"
Oct 29 22:43:27.692163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.692075    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:27.692163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.692108    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220a1952-3291-485e-b6d4-4336b1cd1db3-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:27.692163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.692123    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:27.692163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.692137    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:27.692163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.692149    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:27.692163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.692166    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a1952-3291-485e-b6d4-4336b1cd1db3-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:27.693626 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.692182    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/220a1952-3291-485e-b6d4-4336b1cd1db3-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:27.693626 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.692195    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v94tv\" (UniqueName: \"kubernetes.io/projected/220a1952-3291-485e-b6d4-4336b1cd1db3-kube-api-access-v94tv\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:43:27.928600 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.928561    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"]
Oct 29 22:43:27.941870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:27.941827    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-bkgjl"]
Oct 29 22:43:29.232626 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:29.232587    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="220a1952-3291-485e-b6d4-4336b1cd1db3" path="/var/lib/kubelet/pods/220a1952-3291-485e-b6d4-4336b1cd1db3/volumes"
Oct 29 22:43:30.681839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:30.639635    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:43:42.004503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.004465    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"]
Oct 29 22:43:42.005366 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.005347    4559 cpu_manager.go:414] "containerMap: removing container" podUID="220a1952-3291-485e-b6d4-4336b1cd1db3" containerName="istio-proxy"
Oct 29 22:43:42.005429 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.005370    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="220a1952-3291-485e-b6d4-4336b1cd1db3" containerName="istio-proxy"
Oct 29 22:43:42.005643 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.005626    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="220a1952-3291-485e-b6d4-4336b1cd1db3" containerName="istio-proxy"
Oct 29 22:43:42.009768 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.009751    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.037450 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.037415    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"]
Oct 29 22:43:42.079465 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.079420    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.079853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.079829    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.080072 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.080056    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.080261 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.080247    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.080432 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.080418    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.080578 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.080563    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltmft\" (UniqueName: \"kubernetes.io/projected/e9e758b8-4105-41d2-a2f6-49e874789dfb-kube-api-access-ltmft\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.080797 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.080783    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.081066 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.081050    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e758b8-4105-41d2-a2f6-49e874789dfb-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.081288 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.081272    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.218924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.218045    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.218924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.218103    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.218924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.218182    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.218924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.218287    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.218924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.218361    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.218924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.218402    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltmft\" (UniqueName: \"kubernetes.io/projected/e9e758b8-4105-41d2-a2f6-49e874789dfb-kube-api-access-ltmft\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.218924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.218541    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.218924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.218693    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e758b8-4105-41d2-a2f6-49e874789dfb-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.218924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.218856    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.225854 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.223593    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.225854 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.223945    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.225854 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.224228    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.225854 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.224854    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.225854 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.225125    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.234160 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.232291    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e758b8-4105-41d2-a2f6-49e874789dfb-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.263879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.257840    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.272872 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.272771    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltmft\" (UniqueName: \"kubernetes.io/projected/e9e758b8-4105-41d2-a2f6-49e874789dfb-kube-api-access-ltmft\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.278393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.278350    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.325910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.325873    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:42.683130 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:43:42.683099    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e758b8_4105_41d2_a2f6_49e874789dfb.slice/crio-f23abf36a131293300984d34f895892a286dd958ddad7a9a044e2eb6b99c1523 WatchSource:0}: Error finding container f23abf36a131293300984d34f895892a286dd958ddad7a9a044e2eb6b99c1523: Status 404 returned error can't find the container with id f23abf36a131293300984d34f895892a286dd958ddad7a9a044e2eb6b99c1523
Oct 29 22:43:42.685852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.685822    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"]
Oct 29 22:43:42.686342 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.686312    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:43:42.686391 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.686379    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:43:42.686428 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:42.686409    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:43:43.085942 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:43.085855    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464" event={"ID":"e9e758b8-4105-41d2-a2f6-49e874789dfb","Type":"ContainerStarted","Data":"92894fb099b8ee7948f9a2bdd37440d1017e638ee9ef3775da8af322cadb14b9"}
Oct 29 22:43:43.085942 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:43.085918    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464" event={"ID":"e9e758b8-4105-41d2-a2f6-49e874789dfb","Type":"ContainerStarted","Data":"f23abf36a131293300984d34f895892a286dd958ddad7a9a044e2eb6b99c1523"}
Oct 29 22:43:43.134889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:43.134795    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464" podStartSLOduration=2.134773143 podStartE2EDuration="2.134773143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:43.129412218 +0000 UTC m=+1762.040855773" watchObservedRunningTime="2026-10-29 22:43:43.134773143 +0000 UTC m=+1762.046216698"
Oct 29 22:43:43.326030 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:43.325993    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:43.327848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:43.327821    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 29 22:43:43.327989 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:43.327891    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464" podUID="e9e758b8-4105-41d2-a2f6-49e874789dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 29 22:43:43.704522 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:43:43.704487    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:43:44.190614 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:44.190577    4559 generic.go:356] "Generic (PLEG): container finished" podID="44273e03-eb0c-48e0-82ed-9adfc7badd3d" containerID="305fff2921d2a8374ea20fbb65a2458789e31ffc891310c3c058f9df56884ab7" exitCode=0
Oct 29 22:43:44.191376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:44.191351    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd" event={"ID":"44273e03-eb0c-48e0-82ed-9adfc7badd3d","Type":"ContainerDied","Data":"305fff2921d2a8374ea20fbb65a2458789e31ffc891310c3c058f9df56884ab7"}
Oct 29 22:43:44.191703 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:44.191684    4559 scope.go:122] "RemoveContainer" containerID="305fff2921d2a8374ea20fbb65a2458789e31ffc891310c3c058f9df56884ab7"
Oct 29 22:43:44.193533 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:44.193500    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:43:44.193637 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:44.193572    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:43:44.193637 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:44.193616    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:43:44.326972 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:44.326943    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 29 22:43:44.327202 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:44.327003    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464" podUID="e9e758b8-4105-41d2-a2f6-49e874789dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 29 22:43:44.892590 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:44.892551    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:43:45.202316 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:45.202221    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd" event={"ID":"44273e03-eb0c-48e0-82ed-9adfc7badd3d","Type":"ContainerStarted","Data":"bb7959c54cc9c8a0b46975390a69787da7d8237ed40acb14a3f238c01ac27d39"}
Oct 29 22:43:45.327060 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:45.327023    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 29 22:43:45.327242 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:45.327079    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464" podUID="e9e758b8-4105-41d2-a2f6-49e874789dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 29 22:43:45.584894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:45.584578    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:43:45.593920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:45.593800    4559 patch_prober.go:29] interesting pod/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 29 22:43:45.593920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:45.593890    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd" podUID="44273e03-eb0c-48e0-82ed-9adfc7badd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 29 22:43:46.326868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:46.326835    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 29 22:43:46.327383 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:46.326896    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464" podUID="e9e758b8-4105-41d2-a2f6-49e874789dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 29 22:43:46.578885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:46.577944    4559 patch_prober.go:29] interesting pod/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 29 22:43:46.578885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:46.578002    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd" podUID="44273e03-eb0c-48e0-82ed-9adfc7badd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 29 22:43:47.326783 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:47.326748    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 29 22:43:47.327038 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:47.327013    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464" podUID="e9e758b8-4105-41d2-a2f6-49e874789dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 29 22:43:47.579277 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:47.579184    4559 patch_prober.go:29] interesting pod/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 29 22:43:47.579602 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:47.579560    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd" podUID="44273e03-eb0c-48e0-82ed-9adfc7badd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 29 22:43:48.327016 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:48.326987    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 29 22:43:48.327324 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:48.327298    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464" podUID="e9e758b8-4105-41d2-a2f6-49e874789dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 29 22:43:48.584901 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:48.584037    4559 patch_prober.go:29] interesting pod/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 29 22:43:48.584901 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:48.584133    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd" podUID="44273e03-eb0c-48e0-82ed-9adfc7badd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 29 22:43:49.332773 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:49.328941    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 29 22:43:49.332773 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:49.329009    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464" podUID="e9e758b8-4105-41d2-a2f6-49e874789dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 29 22:43:49.577667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:49.577626    4559 patch_prober.go:29] interesting pod/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 29 22:43:49.577928 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:49.577699    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd" podUID="44273e03-eb0c-48e0-82ed-9adfc7badd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 29 22:43:50.331899 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:50.331800    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:50.332263 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:50.332237    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:50.362837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:50.350854    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:43:50.578782 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:50.578722    4559 patch_prober.go:29] interesting pod/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 29 22:43:50.579227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:50.579195    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd" podUID="44273e03-eb0c-48e0-82ed-9adfc7badd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 29 22:43:51.583759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:51.583715    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:43:51.584217 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:51.584195    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:43:51.584924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:43:51.584899    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:44:13.702068 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:44:13.702017    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:44:19.091268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:19.091230    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:44:19.139801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:19.139764    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"]
Oct 29 22:44:21.592518 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:21.592483    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:44:21.595137 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:21.595109    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:44:30.736386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:30.736327    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"]
Oct 29 22:44:30.737722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:30.737676    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw" podUID="ec0ef457-75e7-432c-8023-3ef04b7dbbda" containerName="istio-proxy" containerID="cri-o://bc0c118ac6eeda6aa744213e8179aec762cf5103094ca057bfa5f2be213c3206" gracePeriod=30
Oct 29 22:44:30.792689 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:30.772891    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"]
Oct 29 22:44:30.792689 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:30.791148    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd" podUID="09ca3404-6dc6-409a-9eab-8bf1c44cc045" containerName="istio-proxy" containerID="cri-o://993408cb4d70b3db27efd789268a6b1a4035a5dabbafea32320dabd76bfddac1" gracePeriod=30
Oct 29 22:44:30.798160 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:30.798128    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"]
Oct 29 22:44:30.798398 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:30.798366    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v" podUID="1a8682c2-1888-4569-9b49-a12f036b968b" containerName="istio-proxy" containerID="cri-o://5304fca4a21d1f6010aca1b3e95b822e26110ee075e68623a8423d251c37a94f" gracePeriod=30
Oct 29 22:44:33.935787 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:33.935739    4559 generic.go:356] "Generic (PLEG): container finished" podID="fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" containerID="234e547bc5821a880a916bd6b684de6dbee12b5377dffafb082c7b20aaf3ea48" exitCode=0
Oct 29 22:44:33.936386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:33.935859    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v" event={"ID":"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6","Type":"ContainerDied","Data":"234e547bc5821a880a916bd6b684de6dbee12b5377dffafb082c7b20aaf3ea48"}
Oct 29 22:44:33.936491 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:33.936472    4559 scope.go:122] "RemoveContainer" containerID="234e547bc5821a880a916bd6b684de6dbee12b5377dffafb082c7b20aaf3ea48"
Oct 29 22:44:33.939255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:33.939217    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:44:33.939502 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:33.939480    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:44:33.939575 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:33.939556    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:44:34.166548 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.166498    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc" podUID="984111f3-9faf-43d2-8554-7bec87910f0b" containerName="main" containerID="cri-o://e299880ae898ec3a794ba36f619c271dfda0c409e1e136874f336b9477bc8dd6" gracePeriod=45
Oct 29 22:44:34.219800 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.219747    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc" podUID="984111f3-9faf-43d2-8554-7bec87910f0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": dial tcp 10.134.0.142:8000: connect: connection refused"
Oct 29 22:44:34.598893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.598569    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:44:34.627890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.626090    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-dshm\") pod \"984111f3-9faf-43d2-8554-7bec87910f0b\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") "
Oct 29 22:44:34.627890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.626158    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-home\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-home\") pod \"984111f3-9faf-43d2-8554-7bec87910f0b\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") "
Oct 29 22:44:34.627890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.626311    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-tmp-dir\") pod \"984111f3-9faf-43d2-8554-7bec87910f0b\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") "
Oct 29 22:44:34.627890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.626369    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/984111f3-9faf-43d2-8554-7bec87910f0b-tls-certs\" (UniqueName: \"kubernetes.io/secret/984111f3-9faf-43d2-8554-7bec87910f0b-tls-certs\") pod \"984111f3-9faf-43d2-8554-7bec87910f0b\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") "
Oct 29 22:44:34.627890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.626500    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-model-cache\") pod \"984111f3-9faf-43d2-8554-7bec87910f0b\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") "
Oct 29 22:44:34.627890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.626542    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984111f3-9faf-43d2-8554-7bec87910f0b-kube-api-access-2587k\" (UniqueName: \"kubernetes.io/projected/984111f3-9faf-43d2-8554-7bec87910f0b-kube-api-access-2587k\") pod \"984111f3-9faf-43d2-8554-7bec87910f0b\" (UID: \"984111f3-9faf-43d2-8554-7bec87910f0b\") "
Oct 29 22:44:34.627890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.626833    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-tmp-dir" pod "984111f3-9faf-43d2-8554-7bec87910f0b" (UID: "984111f3-9faf-43d2-8554-7bec87910f0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:34.627890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.626969    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-home" pod "984111f3-9faf-43d2-8554-7bec87910f0b" (UID: "984111f3-9faf-43d2-8554-7bec87910f0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:34.627890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.626993    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-model-cache" pod "984111f3-9faf-43d2-8554-7bec87910f0b" (UID: "984111f3-9faf-43d2-8554-7bec87910f0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:34.627890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.627725    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:34.627890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.627750    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:34.627890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.627766    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:34.630170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.629325    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984111f3-9faf-43d2-8554-7bec87910f0b-tls-certs" pod "984111f3-9faf-43d2-8554-7bec87910f0b" (UID: "984111f3-9faf-43d2-8554-7bec87910f0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:44:34.630170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.629361    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984111f3-9faf-43d2-8554-7bec87910f0b-kube-api-access-2587k" pod "984111f3-9faf-43d2-8554-7bec87910f0b" (UID: "984111f3-9faf-43d2-8554-7bec87910f0b"). InnerVolumeSpecName "kube-api-access-2587k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:44:34.630484 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.630451    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-dshm" pod "984111f3-9faf-43d2-8554-7bec87910f0b" (UID: "984111f3-9faf-43d2-8554-7bec87910f0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:34.729419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.729381    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/984111f3-9faf-43d2-8554-7bec87910f0b-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:34.729419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.729413    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2587k\" (UniqueName: \"kubernetes.io/projected/984111f3-9faf-43d2-8554-7bec87910f0b-kube-api-access-2587k\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:34.729419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.729424    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/984111f3-9faf-43d2-8554-7bec87910f0b-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:34.948428 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.948390    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v" event={"ID":"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6","Type":"ContainerStarted","Data":"25807341e0d4bbbb160f73b582b3db4f1217baca011e958295ecf5d26d6d1cf0"}
Oct 29 22:44:34.950037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.950007    4559 generic.go:356] "Generic (PLEG): container finished" podID="984111f3-9faf-43d2-8554-7bec87910f0b" containerID="e299880ae898ec3a794ba36f619c271dfda0c409e1e136874f336b9477bc8dd6" exitCode=0
Oct 29 22:44:34.950188 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.950046    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"
Oct 29 22:44:34.950188 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.950087    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc" event={"ID":"984111f3-9faf-43d2-8554-7bec87910f0b","Type":"ContainerDied","Data":"e299880ae898ec3a794ba36f619c271dfda0c409e1e136874f336b9477bc8dd6"}
Oct 29 22:44:34.950188 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.950128    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc" event={"ID":"984111f3-9faf-43d2-8554-7bec87910f0b","Type":"ContainerDied","Data":"76879691a2a802010e22578bbd57f3a6e7a8e22efff16c1d2884dba27ec05c98"}
Oct 29 22:44:34.950188 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.950146    4559 scope.go:122] "RemoveContainer" containerID="e299880ae898ec3a794ba36f619c271dfda0c409e1e136874f336b9477bc8dd6"
Oct 29 22:44:34.964628 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.964599    4559 scope.go:122] "RemoveContainer" containerID="e299880ae898ec3a794ba36f619c271dfda0c409e1e136874f336b9477bc8dd6"
Oct 29 22:44:34.965018 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:44:34.964984    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e299880ae898ec3a794ba36f619c271dfda0c409e1e136874f336b9477bc8dd6\": container with ID starting with e299880ae898ec3a794ba36f619c271dfda0c409e1e136874f336b9477bc8dd6 not found: ID does not exist" containerID="e299880ae898ec3a794ba36f619c271dfda0c409e1e136874f336b9477bc8dd6"
Oct 29 22:44:34.965124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:34.965037    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e299880ae898ec3a794ba36f619c271dfda0c409e1e136874f336b9477bc8dd6"} err="failed to get container status \"e299880ae898ec3a794ba36f619c271dfda0c409e1e136874f336b9477bc8dd6\": rpc error: code = NotFound desc = could not find container \"e299880ae898ec3a794ba36f619c271dfda0c409e1e136874f336b9477bc8dd6\": container with ID starting with e299880ae898ec3a794ba36f619c271dfda0c409e1e136874f336b9477bc8dd6 not found: ID does not exist"
Oct 29 22:44:35.030860 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:35.030789    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"]
Oct 29 22:44:35.040571 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:35.040541    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4d3168/test-model-4d3168-kserve-6778b8584c-2t9nc"]
Oct 29 22:44:35.230290 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:35.230212    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="984111f3-9faf-43d2-8554-7bec87910f0b" path="/var/lib/kubelet/pods/984111f3-9faf-43d2-8554-7bec87910f0b/volumes"
Oct 29 22:44:35.920273 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:35.920245    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:44:35.922172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:35.921854    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 29 22:44:35.922172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:35.921934    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v" podUID="fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 29 22:44:35.996216 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:35.996184    4559 generic.go:356] "Generic (PLEG): container finished" podID="ec0ef457-75e7-432c-8023-3ef04b7dbbda" containerID="bc0c118ac6eeda6aa744213e8179aec762cf5103094ca057bfa5f2be213c3206" exitCode=0
Oct 29 22:44:35.996730 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:35.996303    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw" event={"ID":"ec0ef457-75e7-432c-8023-3ef04b7dbbda","Type":"ContainerDied","Data":"bc0c118ac6eeda6aa744213e8179aec762cf5103094ca057bfa5f2be213c3206"}
Oct 29 22:44:35.998586 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:35.998558    4559 generic.go:356] "Generic (PLEG): container finished" podID="09ca3404-6dc6-409a-9eab-8bf1c44cc045" containerID="993408cb4d70b3db27efd789268a6b1a4035a5dabbafea32320dabd76bfddac1" exitCode=0
Oct 29 22:44:35.998733 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:35.998649    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd" event={"ID":"09ca3404-6dc6-409a-9eab-8bf1c44cc045","Type":"ContainerDied","Data":"993408cb4d70b3db27efd789268a6b1a4035a5dabbafea32320dabd76bfddac1"}
Oct 29 22:44:36.001752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.000528    4559 generic.go:356] "Generic (PLEG): container finished" podID="1a8682c2-1888-4569-9b49-a12f036b968b" containerID="5304fca4a21d1f6010aca1b3e95b822e26110ee075e68623a8423d251c37a94f" exitCode=0
Oct 29 22:44:36.001752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.001672    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v" event={"ID":"1a8682c2-1888-4569-9b49-a12f036b968b","Type":"ContainerDied","Data":"5304fca4a21d1f6010aca1b3e95b822e26110ee075e68623a8423d251c37a94f"}
Oct 29 22:44:36.295841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.294914    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:44:36.385990 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.385904    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-token\" (UniqueName: \"kubernetes.io/projected/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-token\") pod \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") "
Oct 29 22:44:36.386168 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.386003    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-workload-socket\") pod \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") "
Oct 29 22:44:36.386277 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.386241    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istiod-ca-cert\") pod \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") "
Oct 29 22:44:36.386380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.386366    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-workload-certs\") pod \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") "
Oct 29 22:44:36.386525 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.386512    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-data\") pod \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") "
Oct 29 22:44:36.386632 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.386611    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-credential-socket\") pod \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") "
Oct 29 22:44:36.386693 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.386674    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0ef457-75e7-432c-8023-3ef04b7dbbda-kube-api-access-pzzqc\" (UniqueName: \"kubernetes.io/projected/ec0ef457-75e7-432c-8023-3ef04b7dbbda-kube-api-access-pzzqc\") pod \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") "
Oct 29 22:44:36.386782 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.386767    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-podinfo\") pod \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") "
Oct 29 22:44:36.386879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.386866    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-envoy\") pod \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\" (UID: \"ec0ef457-75e7-432c-8023-3ef04b7dbbda\") "
Oct 29 22:44:36.389940 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.389914    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-envoy" pod "ec0ef457-75e7-432c-8023-3ef04b7dbbda" (UID: "ec0ef457-75e7-432c-8023-3ef04b7dbbda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:36.391155 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.391123    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istiod-ca-cert" pod "ec0ef457-75e7-432c-8023-3ef04b7dbbda" (UID: "ec0ef457-75e7-432c-8023-3ef04b7dbbda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:44:36.391386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.391365    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-workload-socket" pod "ec0ef457-75e7-432c-8023-3ef04b7dbbda" (UID: "ec0ef457-75e7-432c-8023-3ef04b7dbbda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:36.392341 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.392313    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-credential-socket" pod "ec0ef457-75e7-432c-8023-3ef04b7dbbda" (UID: "ec0ef457-75e7-432c-8023-3ef04b7dbbda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:36.392539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.392505    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-workload-certs" pod "ec0ef457-75e7-432c-8023-3ef04b7dbbda" (UID: "ec0ef457-75e7-432c-8023-3ef04b7dbbda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:36.394751 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.394720    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-token" pod "ec0ef457-75e7-432c-8023-3ef04b7dbbda" (UID: "ec0ef457-75e7-432c-8023-3ef04b7dbbda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:44:36.395344 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.395321    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-podinfo" pod "ec0ef457-75e7-432c-8023-3ef04b7dbbda" (UID: "ec0ef457-75e7-432c-8023-3ef04b7dbbda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:44:36.395459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.395398    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0ef457-75e7-432c-8023-3ef04b7dbbda-kube-api-access-pzzqc" pod "ec0ef457-75e7-432c-8023-3ef04b7dbbda" (UID: "ec0ef457-75e7-432c-8023-3ef04b7dbbda"). InnerVolumeSpecName "kube-api-access-pzzqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:44:36.398285 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.398257    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-data" pod "ec0ef457-75e7-432c-8023-3ef04b7dbbda" (UID: "ec0ef457-75e7-432c-8023-3ef04b7dbbda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:36.434301 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.431747    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:44:36.434301 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.433968    4559 patch_prober.go:29] interesting pod/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:44:36.434301 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.434022    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6" podUID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:44:36.489232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.488674    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-envoy\") pod \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") "
Oct 29 22:44:36.489232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.488752    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-workload-certs\") pod \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") "
Oct 29 22:44:36.489232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.488783    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-credential-socket\") pod \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") "
Oct 29 22:44:36.489232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.489007    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istiod-ca-cert\") pod \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") "
Oct 29 22:44:36.489232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.489063    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-token\" (UniqueName: \"kubernetes.io/projected/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-token\") pod \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") "
Oct 29 22:44:36.489232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.489172    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ca3404-6dc6-409a-9eab-8bf1c44cc045-kube-api-access-w5nn2\" (UniqueName: \"kubernetes.io/projected/09ca3404-6dc6-409a-9eab-8bf1c44cc045-kube-api-access-w5nn2\") pod \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") "
Oct 29 22:44:36.489232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.489218    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-podinfo\") pod \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") "
Oct 29 22:44:36.489743 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.489299    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-workload-socket\") pod \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") "
Oct 29 22:44:36.489743 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.489358    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-data\") pod \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\" (UID: \"09ca3404-6dc6-409a-9eab-8bf1c44cc045\") "
Oct 29 22:44:36.491248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.490217    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.491248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.490240    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.491248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.490255    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzzqc\" (UniqueName: \"kubernetes.io/projected/ec0ef457-75e7-432c-8023-3ef04b7dbbda-kube-api-access-pzzqc\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.491248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.490273    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.491248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.490286    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.491248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.490298    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.491248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.490312    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.491248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.490325    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0ef457-75e7-432c-8023-3ef04b7dbbda-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.491248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.490337    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0ef457-75e7-432c-8023-3ef04b7dbbda-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.494065 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.494033    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istiod-ca-cert" pod "09ca3404-6dc6-409a-9eab-8bf1c44cc045" (UID: "09ca3404-6dc6-409a-9eab-8bf1c44cc045"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:44:36.500564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.496677    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-data" pod "09ca3404-6dc6-409a-9eab-8bf1c44cc045" (UID: "09ca3404-6dc6-409a-9eab-8bf1c44cc045"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:36.500564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.496727    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-workload-certs" pod "09ca3404-6dc6-409a-9eab-8bf1c44cc045" (UID: "09ca3404-6dc6-409a-9eab-8bf1c44cc045"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:36.500564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.496924    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-credential-socket" pod "09ca3404-6dc6-409a-9eab-8bf1c44cc045" (UID: "09ca3404-6dc6-409a-9eab-8bf1c44cc045"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:36.500564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.497435    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-envoy" pod "09ca3404-6dc6-409a-9eab-8bf1c44cc045" (UID: "09ca3404-6dc6-409a-9eab-8bf1c44cc045"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:36.501407 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.501381    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-workload-socket" pod "09ca3404-6dc6-409a-9eab-8bf1c44cc045" (UID: "09ca3404-6dc6-409a-9eab-8bf1c44cc045"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:36.509751 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.509720    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-podinfo" pod "09ca3404-6dc6-409a-9eab-8bf1c44cc045" (UID: "09ca3404-6dc6-409a-9eab-8bf1c44cc045"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:44:36.510068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.510044    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-token" pod "09ca3404-6dc6-409a-9eab-8bf1c44cc045" (UID: "09ca3404-6dc6-409a-9eab-8bf1c44cc045"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:44:36.513397 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.511955    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ca3404-6dc6-409a-9eab-8bf1c44cc045-kube-api-access-w5nn2" pod "09ca3404-6dc6-409a-9eab-8bf1c44cc045" (UID: "09ca3404-6dc6-409a-9eab-8bf1c44cc045"). InnerVolumeSpecName "kube-api-access-w5nn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:44:36.521119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.521090    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:44:36.592843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.591984    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-istio-envoy\") pod \"1a8682c2-1888-4569-9b49-a12f036b968b\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") "
Oct 29 22:44:36.592843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.592051    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8682c2-1888-4569-9b49-a12f036b968b-kube-api-access-p4qzf\" (UniqueName: \"kubernetes.io/projected/1a8682c2-1888-4569-9b49-a12f036b968b-kube-api-access-p4qzf\") pod \"1a8682c2-1888-4569-9b49-a12f036b968b\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") "
Oct 29 22:44:36.592843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.592170    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a8682c2-1888-4569-9b49-a12f036b968b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8682c2-1888-4569-9b49-a12f036b968b-istio-podinfo\") pod \"1a8682c2-1888-4569-9b49-a12f036b968b\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") "
Oct 29 22:44:36.592843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.592197    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-istio-data\") pod \"1a8682c2-1888-4569-9b49-a12f036b968b\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") "
Oct 29 22:44:36.592843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.592251    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-workload-certs\") pod \"1a8682c2-1888-4569-9b49-a12f036b968b\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") "
Oct 29 22:44:36.592843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.592317    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-credential-socket\") pod \"1a8682c2-1888-4569-9b49-a12f036b968b\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") "
Oct 29 22:44:36.592843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.592374    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-workload-socket\") pod \"1a8682c2-1888-4569-9b49-a12f036b968b\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") "
Oct 29 22:44:36.592843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.592397    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8682c2-1888-4569-9b49-a12f036b968b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8682c2-1888-4569-9b49-a12f036b968b-istiod-ca-cert\") pod \"1a8682c2-1888-4569-9b49-a12f036b968b\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") "
Oct 29 22:44:36.592843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.592533    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8682c2-1888-4569-9b49-a12f036b968b-istio-token\" (UniqueName: \"kubernetes.io/projected/1a8682c2-1888-4569-9b49-a12f036b968b-istio-token\") pod \"1a8682c2-1888-4569-9b49-a12f036b968b\" (UID: \"1a8682c2-1888-4569-9b49-a12f036b968b\") "
Oct 29 22:44:36.594772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.594335    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.594772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.594365    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.594772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.594377    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.594772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.594389    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.594772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.594402    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.594772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.594415    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5nn2\" (UniqueName: \"kubernetes.io/projected/09ca3404-6dc6-409a-9eab-8bf1c44cc045-kube-api-access-w5nn2\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.594772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.594428    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.594772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.594440    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.594772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.594452    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ca3404-6dc6-409a-9eab-8bf1c44cc045-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.595883 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.595624    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-istio-envoy" pod "1a8682c2-1888-4569-9b49-a12f036b968b" (UID: "1a8682c2-1888-4569-9b49-a12f036b968b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:36.595999 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.595910    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-workload-certs" pod "1a8682c2-1888-4569-9b49-a12f036b968b" (UID: "1a8682c2-1888-4569-9b49-a12f036b968b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:36.596893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.596708    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-workload-socket" pod "1a8682c2-1888-4569-9b49-a12f036b968b" (UID: "1a8682c2-1888-4569-9b49-a12f036b968b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:36.597305 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.597039    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-credential-socket" pod "1a8682c2-1888-4569-9b49-a12f036b968b" (UID: "1a8682c2-1888-4569-9b49-a12f036b968b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:36.598027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.598004    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-istio-data" pod "1a8682c2-1888-4569-9b49-a12f036b968b" (UID: "1a8682c2-1888-4569-9b49-a12f036b968b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:36.598239 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.598197    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8682c2-1888-4569-9b49-a12f036b968b-istio-token" pod "1a8682c2-1888-4569-9b49-a12f036b968b" (UID: "1a8682c2-1888-4569-9b49-a12f036b968b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:44:36.598310 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.598248    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8682c2-1888-4569-9b49-a12f036b968b-istiod-ca-cert" pod "1a8682c2-1888-4569-9b49-a12f036b968b" (UID: "1a8682c2-1888-4569-9b49-a12f036b968b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:44:36.598763 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.598735    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8682c2-1888-4569-9b49-a12f036b968b-kube-api-access-p4qzf" pod "1a8682c2-1888-4569-9b49-a12f036b968b" (UID: "1a8682c2-1888-4569-9b49-a12f036b968b"). InnerVolumeSpecName "kube-api-access-p4qzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:44:36.601017 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.600990    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a8682c2-1888-4569-9b49-a12f036b968b-istio-podinfo" pod "1a8682c2-1888-4569-9b49-a12f036b968b" (UID: "1a8682c2-1888-4569-9b49-a12f036b968b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:44:36.704878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.697033    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.704878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.697061    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4qzf\" (UniqueName: \"kubernetes.io/projected/1a8682c2-1888-4569-9b49-a12f036b968b-kube-api-access-p4qzf\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.704878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.697072    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8682c2-1888-4569-9b49-a12f036b968b-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.704878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.697080    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.704878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.697090    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.704878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.697098    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.704878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.697106    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8682c2-1888-4569-9b49-a12f036b968b-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.704878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.697130    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8682c2-1888-4569-9b49-a12f036b968b-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.704878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.697145    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8682c2-1888-4569-9b49-a12f036b968b-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:36.920058 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.920021    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 29 22:44:36.920230 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:36.920092    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v" podUID="fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 29 22:44:37.011027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.010949    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"
Oct 29 22:44:37.011027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.010973    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw" event={"ID":"ec0ef457-75e7-432c-8023-3ef04b7dbbda","Type":"ContainerDied","Data":"50279129eb40d3222d57c0e8f83b055d14ca535ff262976f64b1bab7319c9313"}
Oct 29 22:44:37.011566 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.011066    4559 scope.go:122] "RemoveContainer" containerID="bc0c118ac6eeda6aa744213e8179aec762cf5103094ca057bfa5f2be213c3206"
Oct 29 22:44:37.013644 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.013229    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"
Oct 29 22:44:37.013644 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.013256    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd" event={"ID":"09ca3404-6dc6-409a-9eab-8bf1c44cc045","Type":"ContainerDied","Data":"3697639271afe9c8e1c4af7b9e5be0f344ea70e1717186aa300f53a7a4b1a004"}
Oct 29 22:44:37.015617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.015549    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v" event={"ID":"1a8682c2-1888-4569-9b49-a12f036b968b","Type":"ContainerDied","Data":"254898a4eb2132b988a9e1f039c2d01533322749a074752086df40ddb7de0e76"}
Oct 29 22:44:37.015780 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.015619    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"
Oct 29 22:44:37.048211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.042403    4559 scope.go:122] "RemoveContainer" containerID="993408cb4d70b3db27efd789268a6b1a4035a5dabbafea32320dabd76bfddac1"
Oct 29 22:44:37.091346 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.091321    4559 scope.go:122] "RemoveContainer" containerID="5304fca4a21d1f6010aca1b3e95b822e26110ee075e68623a8423d251c37a94f"
Oct 29 22:44:37.114538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.114495    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"]
Oct 29 22:44:37.118630 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.118601    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r5b9v"]
Oct 29 22:44:37.178397 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.178354    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"]
Oct 29 22:44:37.185331 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.185297    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-r8hqd"]
Oct 29 22:44:37.204387 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.204348    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"]
Oct 29 22:44:37.208611 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.208582    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-z7hjw"]
Oct 29 22:44:37.229423 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.229394    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ca3404-6dc6-409a-9eab-8bf1c44cc045" path="/var/lib/kubelet/pods/09ca3404-6dc6-409a-9eab-8bf1c44cc045/volumes"
Oct 29 22:44:37.230027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.230011    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8682c2-1888-4569-9b49-a12f036b968b" path="/var/lib/kubelet/pods/1a8682c2-1888-4569-9b49-a12f036b968b/volumes"
Oct 29 22:44:37.230538 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.230522    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec0ef457-75e7-432c-8023-3ef04b7dbbda" path="/var/lib/kubelet/pods/ec0ef457-75e7-432c-8023-3ef04b7dbbda/volumes"
Oct 29 22:44:37.686564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.685640    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"]
Oct 29 22:44:37.686564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.685894    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw" podUID="20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" containerName="istio-proxy" containerID="cri-o://deac0201b3c242632a856d6ce23e0a8b86b0a9b2d5bdb8efdbb4398c65886945" gracePeriod=30
Oct 29 22:44:37.920246 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.920197    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 29 22:44:37.920542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:37.920254    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v" podUID="fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 29 22:44:38.183211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.180706    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"]
Oct 29 22:44:38.183211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.181952    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ec0ef457-75e7-432c-8023-3ef04b7dbbda" containerName="istio-proxy"
Oct 29 22:44:38.183211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.181975    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec0ef457-75e7-432c-8023-3ef04b7dbbda" containerName="istio-proxy"
Oct 29 22:44:38.183211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.182002    4559 cpu_manager.go:414] "containerMap: removing container" podUID="09ca3404-6dc6-409a-9eab-8bf1c44cc045" containerName="istio-proxy"
Oct 29 22:44:38.183211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.182010    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ca3404-6dc6-409a-9eab-8bf1c44cc045" containerName="istio-proxy"
Oct 29 22:44:38.183211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.182027    4559 cpu_manager.go:414] "containerMap: removing container" podUID="1a8682c2-1888-4569-9b49-a12f036b968b" containerName="istio-proxy"
Oct 29 22:44:38.183211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.182034    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8682c2-1888-4569-9b49-a12f036b968b" containerName="istio-proxy"
Oct 29 22:44:38.183211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.182065    4559 cpu_manager.go:414] "containerMap: removing container" podUID="984111f3-9faf-43d2-8554-7bec87910f0b" containerName="main"
Oct 29 22:44:38.183211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.182073    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="984111f3-9faf-43d2-8554-7bec87910f0b" containerName="main"
Oct 29 22:44:38.183211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.182341    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8682c2-1888-4569-9b49-a12f036b968b" containerName="istio-proxy"
Oct 29 22:44:38.183211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.182360    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="984111f3-9faf-43d2-8554-7bec87910f0b" containerName="main"
Oct 29 22:44:38.183211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.182383    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec0ef457-75e7-432c-8023-3ef04b7dbbda" containerName="istio-proxy"
Oct 29 22:44:38.183211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.182393    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ca3404-6dc6-409a-9eab-8bf1c44cc045" containerName="istio-proxy"
Oct 29 22:44:38.220799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.190788    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.220799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.193314    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:44:38.220799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.211466    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8147a8\"/\"kube-root-ca.crt\""
Oct 29 22:44:38.220799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.211694    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8147a8\"/\"openshift-service-ca.crt\""
Oct 29 22:44:38.220799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.212334    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8147a8\"/\"test-model-8147a8-kserve-self-signed-certs\""
Oct 29 22:44:38.220799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.212381    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8147a8\"/\"default-dockercfg-nrjg7\""
Oct 29 22:44:38.220799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.213292    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"]
Oct 29 22:44:38.230764 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.230732    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-tmp-dir\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.230957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.230783    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62156571-1397-4073-ab41-ed9b8038f06a-tls-certs\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.230957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.230832    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czdcm\" (UniqueName: \"kubernetes.io/projected/62156571-1397-4073-ab41-ed9b8038f06a-kube-api-access-czdcm\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.230957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.230868    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-dshm\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.231082 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.231036    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-home\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.231133 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.231115    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-model-cache\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.332726 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.332687    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-home\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.332956 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.332789    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-model-cache\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.333024 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.332977    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-tmp-dir\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.333024 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.333018    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62156571-1397-4073-ab41-ed9b8038f06a-tls-certs\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.333133 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.333067    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czdcm\" (UniqueName: \"kubernetes.io/projected/62156571-1397-4073-ab41-ed9b8038f06a-kube-api-access-czdcm\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.333133 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.333123    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-dshm\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.373944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.333925    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-home\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.373944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.334165    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-model-cache\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.373944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.334666    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-tmp-dir\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.373944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.343519    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-dshm\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.385435 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.379859    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czdcm\" (UniqueName: \"kubernetes.io/projected/62156571-1397-4073-ab41-ed9b8038f06a-kube-api-access-czdcm\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.387195 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.386844    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62156571-1397-4073-ab41-ed9b8038f06a-tls-certs\") pod \"test-model-8147a8-kserve-599675c89c-5xjxs\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") " pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.521843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.518511    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:38.920160 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.920110    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 29 22:44:38.920369 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:38.920178    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v" podUID="fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 29 22:44:39.076375 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:39.076338    4559 generic.go:356] "Generic (PLEG): container finished" podID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerID="50397d1c8c03c78af8db4d8ddc218aaca5002c73f22c9a41a7dc9665adccd445" exitCode=0
Oct 29 22:44:39.076567 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:39.076481    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6" event={"ID":"81d39dc4-cb6c-4265-bcb5-4356168496fb","Type":"ContainerDied","Data":"50397d1c8c03c78af8db4d8ddc218aaca5002c73f22c9a41a7dc9665adccd445"}
Oct 29 22:44:39.077328 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:39.077292    4559 scope.go:122] "RemoveContainer" containerID="50397d1c8c03c78af8db4d8ddc218aaca5002c73f22c9a41a7dc9665adccd445"
Oct 29 22:44:39.079503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:39.079468    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:44:39.079605 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:39.079556    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:44:39.079605 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:39.079590    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:44:39.099221 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:39.099166    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"]
Oct 29 22:44:39.100510 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:44:39.100480    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62156571_1397_4073_ab41_ed9b8038f06a.slice/crio-4311b2a9d20f4788ec634d5cc9468c46db36a43c47400f013c0f594d75608639 WatchSource:0}: Error finding container 4311b2a9d20f4788ec634d5cc9468c46db36a43c47400f013c0f594d75608639: Status 404 returned error can't find the container with id 4311b2a9d20f4788ec634d5cc9468c46db36a43c47400f013c0f594d75608639
Oct 29 22:44:39.919983 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:39.919949    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 29 22:44:39.920460 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:39.920004    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v" podUID="fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 29 22:44:40.088617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:40.088580    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6" event={"ID":"81d39dc4-cb6c-4265-bcb5-4356168496fb","Type":"ContainerStarted","Data":"9fac65e28cbef7fcf82d8ff76f94f436a569419a092fdf83b987807148854ddd"}
Oct 29 22:44:40.090658 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:40.090626    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs" event={"ID":"62156571-1397-4073-ab41-ed9b8038f06a","Type":"ContainerStarted","Data":"9ec33251e2a8f1c644f4746032db0ce7cade3841efdb1b520a73828be6abbeda"}
Oct 29 22:44:40.090841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:40.090673    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs" event={"ID":"62156571-1397-4073-ab41-ed9b8038f06a","Type":"ContainerStarted","Data":"4311b2a9d20f4788ec634d5cc9468c46db36a43c47400f013c0f594d75608639"}
Oct 29 22:44:40.183093 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:40.182972    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs" podStartSLOduration=2.182951226 podStartE2EDuration="2.182951226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:40.17869572 +0000 UTC m=+1819.090139271" watchObservedRunningTime="2026-10-29 22:44:40.182951226 +0000 UTC m=+1819.094394781"
Oct 29 22:44:40.225286 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:40.225251    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:44:40.225470 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:40.225448    4559 patch_prober.go:29] interesting pod/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 29 22:44:40.225520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:40.225493    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6" podUID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 29 22:44:40.523393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:40.520971    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:40.919628 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:40.919591    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 29 22:44:40.919858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:40.919643    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v" podUID="fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 29 22:44:41.225003 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:41.224909    4559 patch_prober.go:29] interesting pod/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 29 22:44:41.225003 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:41.224962    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6" podUID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 29 22:44:41.925069 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:41.925036    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:44:41.926341 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:41.926314    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 22:44:42.225459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:42.225372    4559 patch_prober.go:29] interesting pod/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 29 22:44:42.225459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:42.225431    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6" podUID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 29 22:44:43.118098 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.118068    4559 generic.go:356] "Generic (PLEG): container finished" podID="20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" containerID="deac0201b3c242632a856d6ce23e0a8b86b0a9b2d5bdb8efdbb4398c65886945" exitCode=0
Oct 29 22:44:43.118280 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.118137    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw" event={"ID":"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3","Type":"ContainerDied","Data":"deac0201b3c242632a856d6ce23e0a8b86b0a9b2d5bdb8efdbb4398c65886945"}
Oct 29 22:44:43.118280 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.118214    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw" event={"ID":"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3","Type":"ContainerDied","Data":"525ac70383b84c6c154a22d0962c22c27a767f869d558bffe3bec0ffc47a49dd"}
Oct 29 22:44:43.118280 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.118233    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="525ac70383b84c6c154a22d0962c22c27a767f869d558bffe3bec0ffc47a49dd"
Oct 29 22:44:43.120173 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.120156    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:44:43.194652 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.194612    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"]
Oct 29 22:44:43.225839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.225009    4559 patch_prober.go:29] interesting pod/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 29 22:44:43.225839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.225064    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6" podUID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 29 22:44:43.226270 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.225928    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-token\" (UniqueName: \"kubernetes.io/projected/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-token\") pod \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") "
Oct 29 22:44:43.226270 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.225991    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istiod-ca-cert\") pod \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") "
Oct 29 22:44:43.226270 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.226034    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-podinfo\") pod \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") "
Oct 29 22:44:43.226270 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.226070    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-envoy\") pod \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") "
Oct 29 22:44:43.226270 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.226118    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-workload-certs\") pod \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") "
Oct 29 22:44:43.226270 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.226256    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-kube-api-access-6fk95\" (UniqueName: \"kubernetes.io/projected/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-kube-api-access-6fk95\") pod \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") "
Oct 29 22:44:43.226591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.226419    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-credential-socket\") pod \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") "
Oct 29 22:44:43.226591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.226509    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-workload-socket\") pod \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") "
Oct 29 22:44:43.226709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.226599    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-data\") pod \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\" (UID: \"20009952-bbfe-46ae-8ce8-1fbee6b3c8a3\") "
Oct 29 22:44:43.226709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.226681    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istiod-ca-cert" pod "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" (UID: "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:44:43.227386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.227358    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:43.227562 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.227464    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-credential-socket" pod "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" (UID: "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:43.227697 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.227675    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-workload-socket" pod "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" (UID: "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:43.227776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.227746    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-data" pod "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" (UID: "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:43.229179 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.229156    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-podinfo" pod "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" (UID: "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:44:43.230076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.230039    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-envoy" pod "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" (UID: "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:43.230384 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.230362    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-workload-certs" pod "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" (UID: "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:43.232126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.232101    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-kube-api-access-6fk95" pod "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" (UID: "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3"). InnerVolumeSpecName "kube-api-access-6fk95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:44:43.288305 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.288261    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-token" pod "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" (UID: "20009952-bbfe-46ae-8ce8-1fbee6b3c8a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:44:43.345900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.343318    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fk95\" (UniqueName: \"kubernetes.io/projected/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-kube-api-access-6fk95\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:43.345900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.343365    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:43.345900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.343379    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:43.345900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.343394    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:43.345900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.343407    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:43.345900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.343419    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:43.345900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.343431    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:43.345900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:43.343442    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:43.705910 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:44:43.705873    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:44:44.195834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:44.183425    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"
Oct 29 22:44:44.233100 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:44.232611    4559 patch_prober.go:29] interesting pod/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 29 22:44:44.233100 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:44.232672    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6" podUID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 29 22:44:44.332800 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:44.332729    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"]
Oct 29 22:44:44.353535 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:44.353496    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-kw5tw"]
Oct 29 22:44:45.225031 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:45.224998    4559 patch_prober.go:29] interesting pod/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 29 22:44:45.225243 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:45.225050    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6" podUID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 29 22:44:45.230472 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:45.230438    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" path="/var/lib/kubelet/pods/20009952-bbfe-46ae-8ce8-1fbee6b3c8a3/volumes"
Oct 29 22:44:46.228835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:46.225308    4559 patch_prober.go:29] interesting pod/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 29 22:44:46.228835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:46.225379    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6" podUID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 29 22:44:47.241159 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:47.241123    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:44:47.241622 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:47.241597    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:44:47.242614 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:47.242590    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:44:48.519939 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:48.519907    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:48.533623 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:48.533594    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:49.247585 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:49.247555    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:44:53.218302 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.218264    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"]
Oct 29 22:44:53.219169 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.219150    4559 cpu_manager.go:414] "containerMap: removing container" podUID="20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" containerName="istio-proxy"
Oct 29 22:44:53.219169 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.219169    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" containerName="istio-proxy"
Oct 29 22:44:53.219330 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.219320    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="20009952-bbfe-46ae-8ce8-1fbee6b3c8a3" containerName="istio-proxy"
Oct 29 22:44:53.224753 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.224730    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.260659 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.260625    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a5af18\"/\"kube-root-ca.crt\""
Oct 29 22:44:53.261346 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.261320    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"]
Oct 29 22:44:53.274124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.274093    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a5af18\"/\"test-model-a5af18-kserve-self-signed-certs\""
Oct 29 22:44:53.274124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.274119    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a5af18\"/\"default-dockercfg-5ddjd\""
Oct 29 22:44:53.334050 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.334002    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-tmp-dir\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.334259 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.334098    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-model-cache\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.334377 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.334293    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-dshm\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.334439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.334396    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5e69a0-9148-479e-8590-48f3a4848d5b-tls-certs\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.334496 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.334453    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-home\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.334688 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.334659    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69gsk\" (UniqueName: \"kubernetes.io/projected/ea5e69a0-9148-479e-8590-48f3a4848d5b-kube-api-access-69gsk\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.355506 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.355474    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a5af18\"/\"openshift-service-ca.crt\""
Oct 29 22:44:53.436385 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.436343    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5e69a0-9148-479e-8590-48f3a4848d5b-tls-certs\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.436592 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.436406    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-home\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.436592 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.436578    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69gsk\" (UniqueName: \"kubernetes.io/projected/ea5e69a0-9148-479e-8590-48f3a4848d5b-kube-api-access-69gsk\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.436718 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.436666    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-tmp-dir\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.436774 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.436724    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-model-cache\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.436912 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.436892    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-dshm\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.437023 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.436959    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-home\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.437093 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.437068    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-tmp-dir\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.437167 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.437142    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-model-cache\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.439233 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.439209    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-dshm\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.439695 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.439673    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5e69a0-9148-479e-8590-48f3a4848d5b-tls-certs\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.475143 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.475069    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69gsk\" (UniqueName: \"kubernetes.io/projected/ea5e69a0-9148-479e-8590-48f3a4848d5b-kube-api-access-69gsk\") pod \"test-model-a5af18-kserve-c978f998c-dr88f\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") " pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:53.542838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:53.540679    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:54.135507 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:44:54.135477    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea5e69a0_9148_479e_8590_48f3a4848d5b.slice/crio-458018d7a2b896c18f03b5a87e42fe8fcedcd051f37892296544eea89a882f5d WatchSource:0}: Error finding container 458018d7a2b896c18f03b5a87e42fe8fcedcd051f37892296544eea89a882f5d: Status 404 returned error can't find the container with id 458018d7a2b896c18f03b5a87e42fe8fcedcd051f37892296544eea89a882f5d
Oct 29 22:44:54.138461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:54.138427    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"]
Oct 29 22:44:54.247139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:54.247108    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"]
Oct 29 22:44:54.312803 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:54.312764    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"]
Oct 29 22:44:54.358363 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:54.358324    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f" event={"ID":"ea5e69a0-9148-479e-8590-48f3a4848d5b","Type":"ContainerStarted","Data":"6cab014def79e249b5990bf1961165c3b830509f15eab74dd10e45f923f94848"}
Oct 29 22:44:54.358540 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:54.358376    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f" event={"ID":"ea5e69a0-9148-479e-8590-48f3a4848d5b","Type":"ContainerStarted","Data":"458018d7a2b896c18f03b5a87e42fe8fcedcd051f37892296544eea89a882f5d"}
Oct 29 22:44:54.436763 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:54.436703    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f" podStartSLOduration=1.4366859619999999 podStartE2EDuration="1.436685962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:54.432484714 +0000 UTC m=+1833.343928274" watchObservedRunningTime="2026-10-29 22:44:54.436685962 +0000 UTC m=+1833.348129538"
Oct 29 22:44:54.543230 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:54.543197    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:44:55.346868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.344076    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"]
Oct 29 22:44:55.355211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.355153    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.375933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.375905    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e619b6\"/\"kube-root-ca.crt\""
Oct 29 22:44:55.376074 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.375932    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e619b6\"/\"openshift-service-ca.crt\""
Oct 29 22:44:55.386385 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.386359    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e619b6\"/\"default-dockercfg-84rhf\""
Oct 29 22:44:55.387893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.387864    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e619b6\"/\"test-model-e619b6-kserve-self-signed-certs\""
Oct 29 22:44:55.388453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.388436    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"]
Oct 29 22:44:55.397572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.397538    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-dshm\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.397715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.397576    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bbwp\" (UniqueName: \"kubernetes.io/projected/568eefef-fcf7-445b-8dfd-a6b7e79215ad-kube-api-access-2bbwp\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.397715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.397679    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568eefef-fcf7-445b-8dfd-a6b7e79215ad-tls-certs\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.397847 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.397726    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-home\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.398023 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.397998    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-model-cache\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.398290 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.398273    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-tmp-dir\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.499474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.499434    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-dshm\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.499474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.499475    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bbwp\" (UniqueName: \"kubernetes.io/projected/568eefef-fcf7-445b-8dfd-a6b7e79215ad-kube-api-access-2bbwp\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.499773 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.499505    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568eefef-fcf7-445b-8dfd-a6b7e79215ad-tls-certs\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.499773 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.499546    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-home\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.499773 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.499711    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-model-cache\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.499974 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.499959    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-tmp-dir\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.500030 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.499980    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-home\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.500265 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.500180    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-model-cache\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.500265 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.500203    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-tmp-dir\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.502240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.502216    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-dshm\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.502427 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.502409    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568eefef-fcf7-445b-8dfd-a6b7e79215ad-tls-certs\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.524127 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.524098    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bbwp\" (UniqueName: \"kubernetes.io/projected/568eefef-fcf7-445b-8dfd-a6b7e79215ad-kube-api-access-2bbwp\") pod \"test-model-e619b6-kserve-d94d466d5-97rtz\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") " pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:55.674204 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:55.674151    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:56.187302 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:56.187269    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"]
Oct 29 22:44:56.187442 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:44:56.187281    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod568eefef_fcf7_445b_8dfd_a6b7e79215ad.slice/crio-a6757bd0ae227b388b0dbe24a9479913be5a578b2e03222547cf4469c49af54f WatchSource:0}: Error finding container a6757bd0ae227b388b0dbe24a9479913be5a578b2e03222547cf4469c49af54f: Status 404 returned error can't find the container with id a6757bd0ae227b388b0dbe24a9479913be5a578b2e03222547cf4469c49af54f
Oct 29 22:44:56.378498 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:56.378459    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz" event={"ID":"568eefef-fcf7-445b-8dfd-a6b7e79215ad","Type":"ContainerStarted","Data":"444a483475fa6da5bf47c291077bed93c3555b72b11e6130423ea39f5475d7e2"}
Oct 29 22:44:56.378965 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:56.378516    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz" event={"ID":"568eefef-fcf7-445b-8dfd-a6b7e79215ad","Type":"ContainerStarted","Data":"a6757bd0ae227b388b0dbe24a9479913be5a578b2e03222547cf4469c49af54f"}
Oct 29 22:44:56.475092 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:56.474976    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz" podStartSLOduration=1.474956687 podStartE2EDuration="1.474956687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:56.469068022 +0000 UTC m=+1835.380511577" watchObservedRunningTime="2026-10-29 22:44:56.474956687 +0000 UTC m=+1835.386400242"
Oct 29 22:44:56.683836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:56.682942    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:44:58.233623 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.233576    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk" podUID="38887980-8bf2-4efd-8f38-7a7fda65dadf" containerName="main" containerID="cri-o://7e1fad7bbbcb374b8b799830111bb2b034944e654dba1a00d07205bdaae10492" gracePeriod=45
Oct 29 22:44:58.397432 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.397395    4559 generic.go:356] "Generic (PLEG): container finished" podID="38887980-8bf2-4efd-8f38-7a7fda65dadf" containerID="7e1fad7bbbcb374b8b799830111bb2b034944e654dba1a00d07205bdaae10492" exitCode=0
Oct 29 22:44:58.397602 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.397464    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk" event={"ID":"38887980-8bf2-4efd-8f38-7a7fda65dadf","Type":"ContainerDied","Data":"7e1fad7bbbcb374b8b799830111bb2b034944e654dba1a00d07205bdaae10492"}
Oct 29 22:44:58.455209 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.455172    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"]
Oct 29 22:44:58.461443 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.461409    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.491885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.488416    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"]
Oct 29 22:44:58.491885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.488710    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c7d192\"/\"openshift-service-ca.crt\""
Oct 29 22:44:58.493778 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.493753    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c7d192\"/\"kube-root-ca.crt\""
Oct 29 22:44:58.500270 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.500239    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733bd7c3-3efb-4004-bdd9-10e01133855c-tls-certs\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.500410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.500303    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-model-cache\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.500512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.500491    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nvqv\" (UniqueName: \"kubernetes.io/projected/733bd7c3-3efb-4004-bdd9-10e01133855c-kube-api-access-8nvqv\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.500674 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.500658    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-dshm\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.500906 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.500879    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-home\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.500999 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.500969    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-tmp-dir\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.501608 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.501587    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c7d192\"/\"test-model-c7d192-kserve-self-signed-certs\""
Oct 29 22:44:58.501699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.501590    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c7d192\"/\"default-dockercfg-v5dnp\""
Oct 29 22:44:58.571282 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.571251    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:44:58.602621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.602589    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-model-cache\") pod \"38887980-8bf2-4efd-8f38-7a7fda65dadf\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") "
Oct 29 22:44:58.602799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.602639    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-tmp-dir\") pod \"38887980-8bf2-4efd-8f38-7a7fda65dadf\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") "
Oct 29 22:44:58.602799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.602772    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38887980-8bf2-4efd-8f38-7a7fda65dadf-tls-certs\" (UniqueName: \"kubernetes.io/secret/38887980-8bf2-4efd-8f38-7a7fda65dadf-tls-certs\") pod \"38887980-8bf2-4efd-8f38-7a7fda65dadf\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") "
Oct 29 22:44:58.602947 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.602858    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-home\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-home\") pod \"38887980-8bf2-4efd-8f38-7a7fda65dadf\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") "
Oct 29 22:44:58.602947 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.602888    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-model-cache" pod "38887980-8bf2-4efd-8f38-7a7fda65dadf" (UID: "38887980-8bf2-4efd-8f38-7a7fda65dadf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:58.602947 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.602918    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38887980-8bf2-4efd-8f38-7a7fda65dadf-kube-api-access-xjcdv\" (UniqueName: \"kubernetes.io/projected/38887980-8bf2-4efd-8f38-7a7fda65dadf-kube-api-access-xjcdv\") pod \"38887980-8bf2-4efd-8f38-7a7fda65dadf\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") "
Oct 29 22:44:58.603106 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.602957    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-dshm\") pod \"38887980-8bf2-4efd-8f38-7a7fda65dadf\" (UID: \"38887980-8bf2-4efd-8f38-7a7fda65dadf\") "
Oct 29 22:44:58.603106 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.602996    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-tmp-dir" pod "38887980-8bf2-4efd-8f38-7a7fda65dadf" (UID: "38887980-8bf2-4efd-8f38-7a7fda65dadf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:58.603106 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.603086    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-home" pod "38887980-8bf2-4efd-8f38-7a7fda65dadf" (UID: "38887980-8bf2-4efd-8f38-7a7fda65dadf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:58.603323 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.603299    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733bd7c3-3efb-4004-bdd9-10e01133855c-tls-certs\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.603403 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.603335    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-model-cache\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.603467 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.603408    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nvqv\" (UniqueName: \"kubernetes.io/projected/733bd7c3-3efb-4004-bdd9-10e01133855c-kube-api-access-8nvqv\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.603540 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.603490    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-dshm\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.604252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.603632    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-home\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.604252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.603673    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-tmp-dir\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.604252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.603964    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:58.604252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.603985    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:58.604252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.603999    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:58.604252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.604116    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-model-cache\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.604594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.604276    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-tmp-dir\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.604594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.604341    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-home\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.605777 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.605747    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38887980-8bf2-4efd-8f38-7a7fda65dadf-tls-certs" pod "38887980-8bf2-4efd-8f38-7a7fda65dadf" (UID: "38887980-8bf2-4efd-8f38-7a7fda65dadf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:44:58.605913 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.605875    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-dshm" pod "38887980-8bf2-4efd-8f38-7a7fda65dadf" (UID: "38887980-8bf2-4efd-8f38-7a7fda65dadf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:58.605977 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.605920    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38887980-8bf2-4efd-8f38-7a7fda65dadf-kube-api-access-xjcdv" pod "38887980-8bf2-4efd-8f38-7a7fda65dadf" (UID: "38887980-8bf2-4efd-8f38-7a7fda65dadf"). InnerVolumeSpecName "kube-api-access-xjcdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:44:58.606407 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.606390    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733bd7c3-3efb-4004-bdd9-10e01133855c-tls-certs\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.606687 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.606667    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-dshm\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.633276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.633247    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nvqv\" (UniqueName: \"kubernetes.io/projected/733bd7c3-3efb-4004-bdd9-10e01133855c-kube-api-access-8nvqv\") pod \"test-model-c7d192-kserve-775b859f67-kgqhd\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") " pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:58.721831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.711940    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38887980-8bf2-4efd-8f38-7a7fda65dadf-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:58.721831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.711976    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjcdv\" (UniqueName: \"kubernetes.io/projected/38887980-8bf2-4efd-8f38-7a7fda65dadf-kube-api-access-xjcdv\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:58.721831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.711991    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38887980-8bf2-4efd-8f38-7a7fda65dadf-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:44:58.788890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:58.788188    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:44:59.248473 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:59.248403    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26" podUID="bcd744cd-3522-40f3-9ef9-857fb1ab36e9" containerName="payload-processing" containerID="cri-o://dc102aff2c38a8f7790d22e14c4cbca22851ab2d711650fbb3c72972bcb75b91" gracePeriod=25
Oct 29 22:44:59.256132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:59.256097    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"]
Oct 29 22:44:59.256483 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:44:59.256449    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod733bd7c3_3efb_4004_bdd9_10e01133855c.slice/crio-7533038a33fccac5dfe1b3969f590aec88584afb1b35823fa04fa6417223fd3a WatchSource:0}: Error finding container 7533038a33fccac5dfe1b3969f590aec88584afb1b35823fa04fa6417223fd3a: Status 404 returned error can't find the container with id 7533038a33fccac5dfe1b3969f590aec88584afb1b35823fa04fa6417223fd3a
Oct 29 22:44:59.313731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:59.313695    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k" podUID="b1912dca-8e71-4a6c-ac38-934bd5f4a8df" containerName="payload-pre-processing" containerID="cri-o://d5d86c226fb1102a3cefe2bdcb8751ce4d6c2a4cbdb7ffc7165ed7372e0c260b" gracePeriod=25
Oct 29 22:44:59.409909 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:59.409870    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd" event={"ID":"733bd7c3-3efb-4004-bdd9-10e01133855c","Type":"ContainerStarted","Data":"894c7792f40350980fed5a111e9fb92ef2b5a36a413fa65a0a9dc12ee28b34bd"}
Oct 29 22:44:59.410075 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:59.409925    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd" event={"ID":"733bd7c3-3efb-4004-bdd9-10e01133855c","Type":"ContainerStarted","Data":"7533038a33fccac5dfe1b3969f590aec88584afb1b35823fa04fa6417223fd3a"}
Oct 29 22:44:59.411652 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:59.411629    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"
Oct 29 22:44:59.411831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:59.411643    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk" event={"ID":"38887980-8bf2-4efd-8f38-7a7fda65dadf","Type":"ContainerDied","Data":"2f6929fce6edf002ca7f5442521abbebe3606b0398ab5de336ee1974170dd2c6"}
Oct 29 22:44:59.411831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:59.411705    4559 scope.go:122] "RemoveContainer" containerID="7e1fad7bbbcb374b8b799830111bb2b034944e654dba1a00d07205bdaae10492"
Oct 29 22:44:59.474521 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:59.474454    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd" podStartSLOduration=1.474438173 podStartE2EDuration="1.474438173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:59.471440372 +0000 UTC m=+1838.382883937" watchObservedRunningTime="2026-10-29 22:44:59.474438173 +0000 UTC m=+1838.385881763"
Oct 29 22:44:59.516479 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:59.516378    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"]
Oct 29 22:44:59.528841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:59.528773    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c296a3/test-model-c296a3-kserve-56d8758455-v2cnk"]
Oct 29 22:44:59.788738 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:44:59.788645    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:45:00.309841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.309797    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89"]
Oct 29 22:45:00.310634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.310620    4559 cpu_manager.go:414] "containerMap: removing container" podUID="38887980-8bf2-4efd-8f38-7a7fda65dadf" containerName="main"
Oct 29 22:45:00.310684 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.310636    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38887980-8bf2-4efd-8f38-7a7fda65dadf" containerName="main"
Oct 29 22:45:00.310763 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.310754    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="38887980-8bf2-4efd-8f38-7a7fda65dadf" containerName="main"
Oct 29 22:45:00.316127 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.316103    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89"
Oct 29 22:45:00.349674 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.349637    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw"]
Oct 29 22:45:00.353689 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.353654    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:45:00.357834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.357787    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw"
Oct 29 22:45:00.363477 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.363445    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89"]
Oct 29 22:45:00.399945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.399248    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89dxf\" (UniqueName: \"kubernetes.io/projected/450a7f81-0bb4-4c1d-9450-13845587b632-kube-api-access-89dxf\") pod \"maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89\" (UID: \"450a7f81-0bb4-4c1d-9450-13845587b632\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89"
Oct 29 22:45:00.399945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.399477    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/450a7f81-0bb4-4c1d-9450-13845587b632-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89\" (UID: \"450a7f81-0bb4-4c1d-9450-13845587b632\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89"
Oct 29 22:45:00.407275 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.407241    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw"]
Oct 29 22:45:00.462580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.462548    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg"]
Oct 29 22:45:00.468117 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.468089    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w"]
Oct 29 22:45:00.468266 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.468206    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg"
Oct 29 22:45:00.475186 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.473803    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w"
Oct 29 22:45:00.501918 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.501509    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaf0d72-b8d6-400c-a285-374310fdace7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw\" (UID: \"cfaf0d72-b8d6-400c-a285-374310fdace7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw"
Oct 29 22:45:00.501918 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.501735    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/450a7f81-0bb4-4c1d-9450-13845587b632-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89\" (UID: \"450a7f81-0bb4-4c1d-9450-13845587b632\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89"
Oct 29 22:45:00.501918 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.501824    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws5ns\" (UniqueName: \"kubernetes.io/projected/cfaf0d72-b8d6-400c-a285-374310fdace7-kube-api-access-ws5ns\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw\" (UID: \"cfaf0d72-b8d6-400c-a285-374310fdace7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw"
Oct 29 22:45:00.502252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.502067    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89dxf\" (UniqueName: \"kubernetes.io/projected/450a7f81-0bb4-4c1d-9450-13845587b632-kube-api-access-89dxf\") pod \"maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89\" (UID: \"450a7f81-0bb4-4c1d-9450-13845587b632\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89"
Oct 29 22:45:00.503229 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.502965    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/450a7f81-0bb4-4c1d-9450-13845587b632-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89\" (UID: \"450a7f81-0bb4-4c1d-9450-13845587b632\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89"
Oct 29 22:45:00.519848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.517870    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg"]
Oct 29 22:45:00.524448 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.523539    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w"]
Oct 29 22:45:00.530467 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.530431    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr"]
Oct 29 22:45:00.536051 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.536025    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr"
Oct 29 22:45:00.537526 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.537500    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs"]
Oct 29 22:45:00.547729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.547689    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs"
Oct 29 22:45:00.551259 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.551226    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx"]
Oct 29 22:45:00.558261 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.558193    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx"
Oct 29 22:45:00.565634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.565565    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89dxf\" (UniqueName: \"kubernetes.io/projected/450a7f81-0bb4-4c1d-9450-13845587b632-kube-api-access-89dxf\") pod \"maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89\" (UID: \"450a7f81-0bb4-4c1d-9450-13845587b632\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89"
Oct 29 22:45:00.565937 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.565916    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr"]
Oct 29 22:45:00.575502 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.575426    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p"]
Oct 29 22:45:00.621795 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.617322    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb74ed25-e90c-48d8-bfb8-7687f250e690-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w\" (UID: \"bb74ed25-e90c-48d8-bfb8-7687f250e690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w"
Oct 29 22:45:00.621795 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.617491    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws5ns\" (UniqueName: \"kubernetes.io/projected/cfaf0d72-b8d6-400c-a285-374310fdace7-kube-api-access-ws5ns\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw\" (UID: \"cfaf0d72-b8d6-400c-a285-374310fdace7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw"
Oct 29 22:45:00.621795 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.617551    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzx78\" (UniqueName: \"kubernetes.io/projected/bb74ed25-e90c-48d8-bfb8-7687f250e690-kube-api-access-jzx78\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w\" (UID: \"bb74ed25-e90c-48d8-bfb8-7687f250e690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w"
Oct 29 22:45:00.621795 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.617740    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/864f2e20-73e9-4552-a8bf-b44acebb7629-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg\" (UID: \"864f2e20-73e9-4552-a8bf-b44acebb7629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg"
Oct 29 22:45:00.621795 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.617899    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r59qf\" (UniqueName: \"kubernetes.io/projected/864f2e20-73e9-4552-a8bf-b44acebb7629-kube-api-access-r59qf\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg\" (UID: \"864f2e20-73e9-4552-a8bf-b44acebb7629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg"
Oct 29 22:45:00.621795 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.617981    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaf0d72-b8d6-400c-a285-374310fdace7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw\" (UID: \"cfaf0d72-b8d6-400c-a285-374310fdace7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw"
Oct 29 22:45:00.621795 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.618887    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaf0d72-b8d6-400c-a285-374310fdace7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw\" (UID: \"cfaf0d72-b8d6-400c-a285-374310fdace7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw"
Oct 29 22:45:00.629290 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.628292    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs"]
Oct 29 22:45:00.629290 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.628496    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p"
Oct 29 22:45:00.648833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.645862    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2"]
Oct 29 22:45:00.657894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.657763    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws5ns\" (UniqueName: \"kubernetes.io/projected/cfaf0d72-b8d6-400c-a285-374310fdace7-kube-api-access-ws5ns\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw\" (UID: \"cfaf0d72-b8d6-400c-a285-374310fdace7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw"
Oct 29 22:45:00.658143 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.658123    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2"
Oct 29 22:45:00.658716 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.658696    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p"]
Oct 29 22:45:00.661520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.661497    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89"
Oct 29 22:45:00.679736 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.679677    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw"
Oct 29 22:45:00.687045 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.686735    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx"]
Oct 29 22:45:00.699027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.698994    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2"]
Oct 29 22:45:00.726414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.719562    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/864f2e20-73e9-4552-a8bf-b44acebb7629-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg\" (UID: \"864f2e20-73e9-4552-a8bf-b44acebb7629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg"
Oct 29 22:45:00.726414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.720495    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0aee5f-3f63-4968-bb93-aec695b43215-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs\" (UID: \"2a0aee5f-3f63-4968-bb93-aec695b43215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs"
Oct 29 22:45:00.726414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.720572    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59qf\" (UniqueName: \"kubernetes.io/projected/864f2e20-73e9-4552-a8bf-b44acebb7629-kube-api-access-r59qf\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg\" (UID: \"864f2e20-73e9-4552-a8bf-b44acebb7629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg"
Oct 29 22:45:00.726414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.720629    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps2kg\" (UniqueName: \"kubernetes.io/projected/2a0aee5f-3f63-4968-bb93-aec695b43215-kube-api-access-ps2kg\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs\" (UID: \"2a0aee5f-3f63-4968-bb93-aec695b43215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs"
Oct 29 22:45:00.726414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.720785    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb74ed25-e90c-48d8-bfb8-7687f250e690-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w\" (UID: \"bb74ed25-e90c-48d8-bfb8-7687f250e690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w"
Oct 29 22:45:00.726414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.720844    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwzh2\" (UniqueName: \"kubernetes.io/projected/698ca9e5-80e2-4475-a865-389166ac7ec9-kube-api-access-zwzh2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx\" (UID: \"698ca9e5-80e2-4475-a865-389166ac7ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx"
Oct 29 22:45:00.726414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.720883    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/873fd28e-ffe4-4c7b-80d3-132892b4523e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr\" (UID: \"873fd28e-ffe4-4c7b-80d3-132892b4523e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr"
Oct 29 22:45:00.726414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.720966    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68q7q\" (UniqueName: \"kubernetes.io/projected/873fd28e-ffe4-4c7b-80d3-132892b4523e-kube-api-access-68q7q\") pod \"maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr\" (UID: \"873fd28e-ffe4-4c7b-80d3-132892b4523e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr"
Oct 29 22:45:00.726414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.721051    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzx78\" (UniqueName: \"kubernetes.io/projected/bb74ed25-e90c-48d8-bfb8-7687f250e690-kube-api-access-jzx78\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w\" (UID: \"bb74ed25-e90c-48d8-bfb8-7687f250e690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w"
Oct 29 22:45:00.726414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.721114    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/698ca9e5-80e2-4475-a865-389166ac7ec9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx\" (UID: \"698ca9e5-80e2-4475-a865-389166ac7ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx"
Oct 29 22:45:00.726414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.723784    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb74ed25-e90c-48d8-bfb8-7687f250e690-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w\" (UID: \"bb74ed25-e90c-48d8-bfb8-7687f250e690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w"
Oct 29 22:45:00.726414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.720414    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/864f2e20-73e9-4552-a8bf-b44acebb7629-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg\" (UID: \"864f2e20-73e9-4552-a8bf-b44acebb7629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg"
Oct 29 22:45:00.830633 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.830589    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwzh2\" (UniqueName: \"kubernetes.io/projected/698ca9e5-80e2-4475-a865-389166ac7ec9-kube-api-access-zwzh2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx\" (UID: \"698ca9e5-80e2-4475-a865-389166ac7ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx"
Oct 29 22:45:00.844955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.844921    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/873fd28e-ffe4-4c7b-80d3-132892b4523e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr\" (UID: \"873fd28e-ffe4-4c7b-80d3-132892b4523e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr"
Oct 29 22:45:00.845320 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.845297    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68q7q\" (UniqueName: \"kubernetes.io/projected/873fd28e-ffe4-4c7b-80d3-132892b4523e-kube-api-access-68q7q\") pod \"maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr\" (UID: \"873fd28e-ffe4-4c7b-80d3-132892b4523e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr"
Oct 29 22:45:00.845867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.845845    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5015fd-ba70-49ca-88aa-7f4f657ed546-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888565-gfh9p\" (UID: \"db5015fd-ba70-49ca-88aa-7f4f657ed546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p"
Oct 29 22:45:00.846016 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.846003    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmlww\" (UniqueName: \"kubernetes.io/projected/db5015fd-ba70-49ca-88aa-7f4f657ed546-kube-api-access-nmlww\") pod \"maas-api-key-cleanup-29888565-gfh9p\" (UID: \"db5015fd-ba70-49ca-88aa-7f4f657ed546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p"
Oct 29 22:45:00.846132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.846120    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/698ca9e5-80e2-4475-a865-389166ac7ec9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx\" (UID: \"698ca9e5-80e2-4475-a865-389166ac7ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx"
Oct 29 22:45:00.846381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.846361    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0aee5f-3f63-4968-bb93-aec695b43215-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs\" (UID: \"2a0aee5f-3f63-4968-bb93-aec695b43215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs"
Oct 29 22:45:00.846853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.846565    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/873fd28e-ffe4-4c7b-80d3-132892b4523e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr\" (UID: \"873fd28e-ffe4-4c7b-80d3-132892b4523e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr"
Oct 29 22:45:00.846853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.845657    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r59qf\" (UniqueName: \"kubernetes.io/projected/864f2e20-73e9-4552-a8bf-b44acebb7629-kube-api-access-r59qf\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg\" (UID: \"864f2e20-73e9-4552-a8bf-b44acebb7629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg"
Oct 29 22:45:00.846853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.846578    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps2kg\" (UniqueName: \"kubernetes.io/projected/2a0aee5f-3f63-4968-bb93-aec695b43215-kube-api-access-ps2kg\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs\" (UID: \"2a0aee5f-3f63-4968-bb93-aec695b43215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs"
Oct 29 22:45:00.846853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.842802    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzx78\" (UniqueName: \"kubernetes.io/projected/bb74ed25-e90c-48d8-bfb8-7687f250e690-kube-api-access-jzx78\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w\" (UID: \"bb74ed25-e90c-48d8-bfb8-7687f250e690\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w"
Oct 29 22:45:00.847655 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.847342    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254d613c-ac01-4d64-90d2-4c4630a1d9f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2\" (UID: \"254d613c-ac01-4d64-90d2-4c4630a1d9f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2"
Oct 29 22:45:00.847655 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.847395    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcvvj\" (UniqueName: \"kubernetes.io/projected/254d613c-ac01-4d64-90d2-4c4630a1d9f1-kube-api-access-zcvvj\") pod \"maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2\" (UID: \"254d613c-ac01-4d64-90d2-4c4630a1d9f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2"
Oct 29 22:45:00.847655 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.847589    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0aee5f-3f63-4968-bb93-aec695b43215-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs\" (UID: \"2a0aee5f-3f63-4968-bb93-aec695b43215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs"
Oct 29 22:45:00.848156 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.848093    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/698ca9e5-80e2-4475-a865-389166ac7ec9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx\" (UID: \"698ca9e5-80e2-4475-a865-389166ac7ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx"
Oct 29 22:45:00.863905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.863875    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwzh2\" (UniqueName: \"kubernetes.io/projected/698ca9e5-80e2-4475-a865-389166ac7ec9-kube-api-access-zwzh2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx\" (UID: \"698ca9e5-80e2-4475-a865-389166ac7ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx"
Oct 29 22:45:00.874436 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.871428    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68q7q\" (UniqueName: \"kubernetes.io/projected/873fd28e-ffe4-4c7b-80d3-132892b4523e-kube-api-access-68q7q\") pod \"maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr\" (UID: \"873fd28e-ffe4-4c7b-80d3-132892b4523e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr"
Oct 29 22:45:00.875488 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.875456    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps2kg\" (UniqueName: \"kubernetes.io/projected/2a0aee5f-3f63-4968-bb93-aec695b43215-kube-api-access-ps2kg\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs\" (UID: \"2a0aee5f-3f63-4968-bb93-aec695b43215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs"
Oct 29 22:45:00.926351 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.926324    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs"
Oct 29 22:45:00.947928 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.947896    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx"
Oct 29 22:45:00.948917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.948886    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254d613c-ac01-4d64-90d2-4c4630a1d9f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2\" (UID: \"254d613c-ac01-4d64-90d2-4c4630a1d9f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2"
Oct 29 22:45:00.949016 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.948948    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcvvj\" (UniqueName: \"kubernetes.io/projected/254d613c-ac01-4d64-90d2-4c4630a1d9f1-kube-api-access-zcvvj\") pod \"maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2\" (UID: \"254d613c-ac01-4d64-90d2-4c4630a1d9f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2"
Oct 29 22:45:00.949223 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.949198    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5015fd-ba70-49ca-88aa-7f4f657ed546-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888565-gfh9p\" (UID: \"db5015fd-ba70-49ca-88aa-7f4f657ed546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p"
Oct 29 22:45:00.949302 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.949258    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmlww\" (UniqueName: \"kubernetes.io/projected/db5015fd-ba70-49ca-88aa-7f4f657ed546-kube-api-access-nmlww\") pod \"maas-api-key-cleanup-29888565-gfh9p\" (UID: \"db5015fd-ba70-49ca-88aa-7f4f657ed546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p"
Oct 29 22:45:00.949561 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.949539    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254d613c-ac01-4d64-90d2-4c4630a1d9f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2\" (UID: \"254d613c-ac01-4d64-90d2-4c4630a1d9f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2"
Oct 29 22:45:00.950379 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.950355    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5015fd-ba70-49ca-88aa-7f4f657ed546-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888565-gfh9p\" (UID: \"db5015fd-ba70-49ca-88aa-7f4f657ed546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p"
Oct 29 22:45:00.971495 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.971462    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmlww\" (UniqueName: \"kubernetes.io/projected/db5015fd-ba70-49ca-88aa-7f4f657ed546-kube-api-access-nmlww\") pod \"maas-api-key-cleanup-29888565-gfh9p\" (UID: \"db5015fd-ba70-49ca-88aa-7f4f657ed546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p"
Oct 29 22:45:00.973502 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.973471    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcvvj\" (UniqueName: \"kubernetes.io/projected/254d613c-ac01-4d64-90d2-4c4630a1d9f1-kube-api-access-zcvvj\") pod \"maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2\" (UID: \"254d613c-ac01-4d64-90d2-4c4630a1d9f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2"
Oct 29 22:45:00.993973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:00.993929    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p"
Oct 29 22:45:01.107266 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:01.107173    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg"
Oct 29 22:45:01.107891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:01.107865    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w"
Oct 29 22:45:01.119366 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:01.119264    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2"
Oct 29 22:45:01.137169 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:45:01.137100    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450a7f81_0bb4_4c1d_9450_13845587b632.slice/crio-bee9f9d11f93ffae1893ebff2ab50c3b1e618abe8aaa4de847c0c72a3a0e3f0a WatchSource:0}: Error finding container bee9f9d11f93ffae1893ebff2ab50c3b1e618abe8aaa4de847c0c72a3a0e3f0a: Status 404 returned error can't find the container with id bee9f9d11f93ffae1893ebff2ab50c3b1e618abe8aaa4de847c0c72a3a0e3f0a
Oct 29 22:45:01.152859 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:01.152275    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89"]
Oct 29 22:45:01.159155 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:01.158657    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw"]
Oct 29 22:45:01.159905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:01.159870    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr"
Oct 29 22:45:01.172023 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:45:01.171987    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfaf0d72_b8d6_400c_a285_374310fdace7.slice/crio-525425b86c24e711ca8dfdcdcca5e139adcedc7f2b5c94d6a9421e314a8a6ffb WatchSource:0}: Error finding container 525425b86c24e711ca8dfdcdcca5e139adcedc7f2b5c94d6a9421e314a8a6ffb: Status 404 returned error can't find the container with id 525425b86c24e711ca8dfdcdcca5e139adcedc7f2b5c94d6a9421e314a8a6ffb
Oct 29 22:45:01.386164 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:01.385506    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38887980-8bf2-4efd-8f38-7a7fda65dadf" path="/var/lib/kubelet/pods/38887980-8bf2-4efd-8f38-7a7fda65dadf/volumes"
Oct 29 22:45:01.386164 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:01.386072    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx"]
Oct 29 22:45:01.390697 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:01.390243    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs"]
Oct 29 22:45:01.430518 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:01.430487    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p"]
Oct 29 22:45:01.433543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:01.433516    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89" event={"ID":"450a7f81-0bb4-4c1d-9450-13845587b632","Type":"ContainerStarted","Data":"bee9f9d11f93ffae1893ebff2ab50c3b1e618abe8aaa4de847c0c72a3a0e3f0a"}
Oct 29 22:45:01.435259 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:01.435226    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw" event={"ID":"cfaf0d72-b8d6-400c-a285-374310fdace7","Type":"ContainerStarted","Data":"525425b86c24e711ca8dfdcdcca5e139adcedc7f2b5c94d6a9421e314a8a6ffb"}
Oct 29 22:45:01.439727 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:45:01.439685    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698ca9e5_80e2_4475_a865_389166ac7ec9.slice/crio-48279e187e58c8342284da88d5bad21b90a5a8adcc869296ea590bd2b77f47ce WatchSource:0}: Error finding container 48279e187e58c8342284da88d5bad21b90a5a8adcc869296ea590bd2b77f47ce: Status 404 returned error can't find the container with id 48279e187e58c8342284da88d5bad21b90a5a8adcc869296ea590bd2b77f47ce
Oct 29 22:45:01.440685 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:45:01.440656    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0aee5f_3f63_4968_bb93_aec695b43215.slice/crio-ddc0d8cdb703c8e9dfb2f65495419bc9123cda1b556f8c62ce6c568841f4f732 WatchSource:0}: Error finding container ddc0d8cdb703c8e9dfb2f65495419bc9123cda1b556f8c62ce6c568841f4f732: Status 404 returned error can't find the container with id ddc0d8cdb703c8e9dfb2f65495419bc9123cda1b556f8c62ce6c568841f4f732
Oct 29 22:45:01.442051 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:45:01.442028    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5015fd_ba70_49ca_88aa_7f4f657ed546.slice/crio-17f1e1461f8da8ca1977a8f6281c966f783d79a58d8cd6016ae6fe084dcb55f0 WatchSource:0}: Error finding container 17f1e1461f8da8ca1977a8f6281c966f783d79a58d8cd6016ae6fe084dcb55f0: Status 404 returned error can't find the container with id 17f1e1461f8da8ca1977a8f6281c966f783d79a58d8cd6016ae6fe084dcb55f0
Oct 29 22:45:01.639712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:01.638975    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr"]
Oct 29 22:45:01.645797 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:45:01.645761    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873fd28e_ffe4_4c7b_80d3_132892b4523e.slice/crio-34be0641d777f7fec0e842e2ed89b87090655006e50b3c678faec9a0fca5d9ce WatchSource:0}: Error finding container 34be0641d777f7fec0e842e2ed89b87090655006e50b3c678faec9a0fca5d9ce: Status 404 returned error can't find the container with id 34be0641d777f7fec0e842e2ed89b87090655006e50b3c678faec9a0fca5d9ce
Oct 29 22:45:01.746357 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:01.746324    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg"]
Oct 29 22:45:01.746793 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:45:01.746770    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864f2e20_73e9_4552_a8bf_b44acebb7629.slice/crio-f235e07d484f819a178c5bb686dc6241526bed87b6fc27ba10eb972b0e59b3de WatchSource:0}: Error finding container f235e07d484f819a178c5bb686dc6241526bed87b6fc27ba10eb972b0e59b3de: Status 404 returned error can't find the container with id f235e07d484f819a178c5bb686dc6241526bed87b6fc27ba10eb972b0e59b3de
Oct 29 22:45:01.761346 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:01.761318    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w"]
Oct 29 22:45:01.763147 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:45:01.763123    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254d613c_ac01_4d64_90d2_4c4630a1d9f1.slice/crio-2db6bf1794f653d5c07579338943ac5d8a22df90765cbb83b122177c8a8bbbcc WatchSource:0}: Error finding container 2db6bf1794f653d5c07579338943ac5d8a22df90765cbb83b122177c8a8bbbcc: Status 404 returned error can't find the container with id 2db6bf1794f653d5c07579338943ac5d8a22df90765cbb83b122177c8a8bbbcc
Oct 29 22:45:01.766498 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:45:01.766475    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb74ed25_e90c_48d8_bfb8_7687f250e690.slice/crio-4b6313c9feb885ff936c22a02c0363150d26496ed6c1f4d2bfa18b45035723a3 WatchSource:0}: Error finding container 4b6313c9feb885ff936c22a02c0363150d26496ed6c1f4d2bfa18b45035723a3: Status 404 returned error can't find the container with id 4b6313c9feb885ff936c22a02c0363150d26496ed6c1f4d2bfa18b45035723a3
Oct 29 22:45:01.767454 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:01.767432    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2"]
Oct 29 22:45:02.454116 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.454084    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w" event={"ID":"bb74ed25-e90c-48d8-bfb8-7687f250e690","Type":"ContainerStarted","Data":"4b6313c9feb885ff936c22a02c0363150d26496ed6c1f4d2bfa18b45035723a3"}
Oct 29 22:45:02.456183 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.456149    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89" event={"ID":"450a7f81-0bb4-4c1d-9450-13845587b632","Type":"ContainerStarted","Data":"48d27cf7a62aa17d63c88933f123e3e56b8912d2eaabcfa46b88e7a72a1d2ea9"}
Oct 29 22:45:02.458340 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.458305    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs" event={"ID":"2a0aee5f-3f63-4968-bb93-aec695b43215","Type":"ContainerStarted","Data":"1f33c705cb4feb9b0736f534c0370500dac5654f9a4721d6f943658f5181c9e1"}
Oct 29 22:45:02.458461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.458369    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs" event={"ID":"2a0aee5f-3f63-4968-bb93-aec695b43215","Type":"ContainerStarted","Data":"ddc0d8cdb703c8e9dfb2f65495419bc9123cda1b556f8c62ce6c568841f4f732"}
Oct 29 22:45:02.460157 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.460131    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p" event={"ID":"db5015fd-ba70-49ca-88aa-7f4f657ed546","Type":"ContainerStarted","Data":"34bbad1fde0096f8a9b1215842f5c506522f0a362a66da9abe2fe612dc3e798d"}
Oct 29 22:45:02.460278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.460179    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p" event={"ID":"db5015fd-ba70-49ca-88aa-7f4f657ed546","Type":"ContainerStarted","Data":"17f1e1461f8da8ca1977a8f6281c966f783d79a58d8cd6016ae6fe084dcb55f0"}
Oct 29 22:45:02.461945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.461920    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr" event={"ID":"873fd28e-ffe4-4c7b-80d3-132892b4523e","Type":"ContainerStarted","Data":"f2ae78db83ec203b386bbc61dd396a293b033c55aa59484d90f8528663917f7e"}
Oct 29 22:45:02.462044 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.461965    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr" event={"ID":"873fd28e-ffe4-4c7b-80d3-132892b4523e","Type":"ContainerStarted","Data":"34be0641d777f7fec0e842e2ed89b87090655006e50b3c678faec9a0fca5d9ce"}
Oct 29 22:45:02.463366 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.463343    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg" event={"ID":"864f2e20-73e9-4552-a8bf-b44acebb7629","Type":"ContainerStarted","Data":"f235e07d484f819a178c5bb686dc6241526bed87b6fc27ba10eb972b0e59b3de"}
Oct 29 22:45:02.465038 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.465014    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx" event={"ID":"698ca9e5-80e2-4475-a865-389166ac7ec9","Type":"ContainerStarted","Data":"27ebd93773f47bab573fdb2842318c4c55a94101abe8aca03a1d7df50d573d37"}
Oct 29 22:45:02.465138 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.465057    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx" event={"ID":"698ca9e5-80e2-4475-a865-389166ac7ec9","Type":"ContainerStarted","Data":"48279e187e58c8342284da88d5bad21b90a5a8adcc869296ea590bd2b77f47ce"}
Oct 29 22:45:02.466714 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.466691    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw" event={"ID":"cfaf0d72-b8d6-400c-a285-374310fdace7","Type":"ContainerStarted","Data":"bc46eb21d3d751535df554bb41af733310109a19e055b2686f3100b4cbe02d21"}
Oct 29 22:45:02.466980 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.466963    4559 scope.go:122] "RemoveContainer" containerID="bc46eb21d3d751535df554bb41af733310109a19e055b2686f3100b4cbe02d21"
Oct 29 22:45:02.468161 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.468137    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2" event={"ID":"254d613c-ac01-4d64-90d2-4c4630a1d9f1","Type":"ContainerStarted","Data":"2db6bf1794f653d5c07579338943ac5d8a22df90765cbb83b122177c8a8bbbcc"}
Oct 29 22:45:02.500995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.500948    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89" podStartSLOduration=2.500933951 podStartE2EDuration="2.500933951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.495060018 +0000 UTC m=+1841.406503574" watchObservedRunningTime="2026-10-29 22:45:02.500933951 +0000 UTC m=+1841.412377535"
Oct 29 22:45:02.618945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.614371    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr" podStartSLOduration=2.614348741 podStartE2EDuration="2.614348741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.552287304 +0000 UTC m=+1841.463730859" watchObservedRunningTime="2026-10-29 22:45:02.614348741 +0000 UTC m=+1841.525792300"
Oct 29 22:45:02.618945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.618615    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx" podStartSLOduration=2.618596369 podStartE2EDuration="2.618596369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.600496423 +0000 UTC m=+1841.511939978" watchObservedRunningTime="2026-10-29 22:45:02.618596369 +0000 UTC m=+1841.530039925"
Oct 29 22:45:02.708406 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.708342    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p" podStartSLOduration=2.708322953 podStartE2EDuration="2.708322953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.644681091 +0000 UTC m=+1841.556124645" watchObservedRunningTime="2026-10-29 22:45:02.708322953 +0000 UTC m=+1841.619766507"
Oct 29 22:45:02.718413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:02.718101    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs" podStartSLOduration=2.718085641 podStartE2EDuration="2.718085641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.687800925 +0000 UTC m=+1841.599244481" watchObservedRunningTime="2026-10-29 22:45:02.718085641 +0000 UTC m=+1841.629529195"
Oct 29 22:45:03.482404 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.482308    4559 generic.go:356] "Generic (PLEG): container finished" podID="873fd28e-ffe4-4c7b-80d3-132892b4523e" containerID="f2ae78db83ec203b386bbc61dd396a293b033c55aa59484d90f8528663917f7e" exitCode=60
Oct 29 22:45:03.482876 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.482398    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr" event={"ID":"873fd28e-ffe4-4c7b-80d3-132892b4523e","Type":"ContainerDied","Data":"f2ae78db83ec203b386bbc61dd396a293b033c55aa59484d90f8528663917f7e"}
Oct 29 22:45:03.482876 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.482732    4559 scope.go:122] "RemoveContainer" containerID="f2ae78db83ec203b386bbc61dd396a293b033c55aa59484d90f8528663917f7e"
Oct 29 22:45:03.484374 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.484352    4559 generic.go:356] "Generic (PLEG): container finished" podID="864f2e20-73e9-4552-a8bf-b44acebb7629" containerID="4e01228a8ff9aecb463a84dcb7fb15dc12b7353e05accd0e6437928b61a01dd2" exitCode=60
Oct 29 22:45:03.484486 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.484426    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg" event={"ID":"864f2e20-73e9-4552-a8bf-b44acebb7629","Type":"ContainerDied","Data":"4e01228a8ff9aecb463a84dcb7fb15dc12b7353e05accd0e6437928b61a01dd2"}
Oct 29 22:45:03.484750 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.484731    4559 scope.go:122] "RemoveContainer" containerID="4e01228a8ff9aecb463a84dcb7fb15dc12b7353e05accd0e6437928b61a01dd2"
Oct 29 22:45:03.486544 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.486494    4559 generic.go:356] "Generic (PLEG): container finished" podID="698ca9e5-80e2-4475-a865-389166ac7ec9" containerID="27ebd93773f47bab573fdb2842318c4c55a94101abe8aca03a1d7df50d573d37" exitCode=60
Oct 29 22:45:03.486602 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.486563    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx" event={"ID":"698ca9e5-80e2-4475-a865-389166ac7ec9","Type":"ContainerDied","Data":"27ebd93773f47bab573fdb2842318c4c55a94101abe8aca03a1d7df50d573d37"}
Oct 29 22:45:03.486978 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.486898    4559 scope.go:122] "RemoveContainer" containerID="27ebd93773f47bab573fdb2842318c4c55a94101abe8aca03a1d7df50d573d37"
Oct 29 22:45:03.488919 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.488898    4559 generic.go:356] "Generic (PLEG): container finished" podID="cfaf0d72-b8d6-400c-a285-374310fdace7" containerID="bc46eb21d3d751535df554bb41af733310109a19e055b2686f3100b4cbe02d21" exitCode=60
Oct 29 22:45:03.488979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.488963    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw" event={"ID":"cfaf0d72-b8d6-400c-a285-374310fdace7","Type":"ContainerDied","Data":"bc46eb21d3d751535df554bb41af733310109a19e055b2686f3100b4cbe02d21"}
Oct 29 22:45:03.489043 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.489006    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw" event={"ID":"cfaf0d72-b8d6-400c-a285-374310fdace7","Type":"ContainerStarted","Data":"c0139ffe4206d61c20c334e1debafe7595628a5cdd70ebf4f90ca952797cfaa2"}
Oct 29 22:45:03.490546 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.490523    4559 generic.go:356] "Generic (PLEG): container finished" podID="254d613c-ac01-4d64-90d2-4c4630a1d9f1" containerID="9197e0eec5aa827c894c095afe6732bf9aa976621f8d69e0020401577c344ede" exitCode=60
Oct 29 22:45:03.490629 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.490594    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2" event={"ID":"254d613c-ac01-4d64-90d2-4c4630a1d9f1","Type":"ContainerDied","Data":"9197e0eec5aa827c894c095afe6732bf9aa976621f8d69e0020401577c344ede"}
Oct 29 22:45:03.490855 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.490800    4559 scope.go:122] "RemoveContainer" containerID="9197e0eec5aa827c894c095afe6732bf9aa976621f8d69e0020401577c344ede"
Oct 29 22:45:03.492256 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.492236    4559 generic.go:356] "Generic (PLEG): container finished" podID="bb74ed25-e90c-48d8-bfb8-7687f250e690" containerID="2341a6db2c554a46ef2b96a62e53f7fc26a22b2321ab804bd48e68caca819b00" exitCode=60
Oct 29 22:45:03.492337 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.492318    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w" event={"ID":"bb74ed25-e90c-48d8-bfb8-7687f250e690","Type":"ContainerDied","Data":"2341a6db2c554a46ef2b96a62e53f7fc26a22b2321ab804bd48e68caca819b00"}
Oct 29 22:45:03.492519 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.492501    4559 scope.go:122] "RemoveContainer" containerID="2341a6db2c554a46ef2b96a62e53f7fc26a22b2321ab804bd48e68caca819b00"
Oct 29 22:45:03.500304 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.500211    4559 generic.go:356] "Generic (PLEG): container finished" podID="450a7f81-0bb4-4c1d-9450-13845587b632" containerID="48d27cf7a62aa17d63c88933f123e3e56b8912d2eaabcfa46b88e7a72a1d2ea9" exitCode=60
Oct 29 22:45:03.500304 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.500245    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89" event={"ID":"450a7f81-0bb4-4c1d-9450-13845587b632","Type":"ContainerDied","Data":"48d27cf7a62aa17d63c88933f123e3e56b8912d2eaabcfa46b88e7a72a1d2ea9"}
Oct 29 22:45:03.500539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.500521    4559 scope.go:122] "RemoveContainer" containerID="48d27cf7a62aa17d63c88933f123e3e56b8912d2eaabcfa46b88e7a72a1d2ea9"
Oct 29 22:45:03.502303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.502278    4559 generic.go:356] "Generic (PLEG): container finished" podID="2a0aee5f-3f63-4968-bb93-aec695b43215" containerID="1f33c705cb4feb9b0736f534c0370500dac5654f9a4721d6f943658f5181c9e1" exitCode=60
Oct 29 22:45:03.502377 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.502355    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs" event={"ID":"2a0aee5f-3f63-4968-bb93-aec695b43215","Type":"ContainerDied","Data":"1f33c705cb4feb9b0736f534c0370500dac5654f9a4721d6f943658f5181c9e1"}
Oct 29 22:45:03.502591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.502574    4559 scope.go:122] "RemoveContainer" containerID="1f33c705cb4feb9b0736f534c0370500dac5654f9a4721d6f943658f5181c9e1"
Oct 29 22:45:03.504174 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.504152    4559 generic.go:356] "Generic (PLEG): container finished" podID="db5015fd-ba70-49ca-88aa-7f4f657ed546" containerID="34bbad1fde0096f8a9b1215842f5c506522f0a362a66da9abe2fe612dc3e798d" exitCode=60
Oct 29 22:45:03.504269 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.504205    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p" event={"ID":"db5015fd-ba70-49ca-88aa-7f4f657ed546","Type":"ContainerDied","Data":"34bbad1fde0096f8a9b1215842f5c506522f0a362a66da9abe2fe612dc3e798d"}
Oct 29 22:45:03.504499 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.504481    4559 scope.go:122] "RemoveContainer" containerID="34bbad1fde0096f8a9b1215842f5c506522f0a362a66da9abe2fe612dc3e798d"
Oct 29 22:45:03.543153 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.543125    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:45:03.557172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.557143    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:45:03.571841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:03.571775    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw" podStartSLOduration=3.57175973 podStartE2EDuration="3.57175973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.5690595 +0000 UTC m=+1842.480503055" watchObservedRunningTime="2026-10-29 22:45:03.57175973 +0000 UTC m=+1842.483203283"
Oct 29 22:45:04.516161 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:04.516124    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr" event={"ID":"873fd28e-ffe4-4c7b-80d3-132892b4523e","Type":"ContainerStarted","Data":"43c7c4ccf793186e16d669913abdb3dc256f528990a54d82846871b467382a21"}
Oct 29 22:45:04.518332 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:04.518304    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg" event={"ID":"864f2e20-73e9-4552-a8bf-b44acebb7629","Type":"ContainerStarted","Data":"e62ebe3b6bddd6046a7b948912dd41b15259bf2b9f1d589e6234a67a001de1b0"}
Oct 29 22:45:04.520424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:04.520399    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx" event={"ID":"698ca9e5-80e2-4475-a865-389166ac7ec9","Type":"ContainerStarted","Data":"dd10e2bc6751650233d89a110cd512a1f0846cdb57bf6788fdb6285ca9ca4e9c"}
Oct 29 22:45:04.522501 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:04.522466    4559 generic.go:356] "Generic (PLEG): container finished" podID="cfaf0d72-b8d6-400c-a285-374310fdace7" containerID="c0139ffe4206d61c20c334e1debafe7595628a5cdd70ebf4f90ca952797cfaa2" exitCode=60
Oct 29 22:45:04.522639 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:04.522532    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw" event={"ID":"cfaf0d72-b8d6-400c-a285-374310fdace7","Type":"ContainerDied","Data":"c0139ffe4206d61c20c334e1debafe7595628a5cdd70ebf4f90ca952797cfaa2"}
Oct 29 22:45:04.522639 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:04.522594    4559 scope.go:122] "RemoveContainer" containerID="bc46eb21d3d751535df554bb41af733310109a19e055b2686f3100b4cbe02d21"
Oct 29 22:45:04.522935 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:04.522910    4559 scope.go:122] "RemoveContainer" containerID="c0139ffe4206d61c20c334e1debafe7595628a5cdd70ebf4f90ca952797cfaa2"
Oct 29 22:45:04.523300 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:04.523280    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw_odh-ai-gateway-infra(cfaf0d72-b8d6-400c-a285-374310fdace7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw" podUID="cfaf0d72-b8d6-400c-a285-374310fdace7"
Oct 29 22:45:04.524614 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:04.524588    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2" event={"ID":"254d613c-ac01-4d64-90d2-4c4630a1d9f1","Type":"ContainerStarted","Data":"089171882c9dd74091640c95221df23193bbbfdf227986b42f4a4337f4251f63"}
Oct 29 22:45:04.526530 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:04.526511    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w" event={"ID":"bb74ed25-e90c-48d8-bfb8-7687f250e690","Type":"ContainerStarted","Data":"a31076e1163e36ce50059c341dde74a7694aa12d340d19bbe7d99fad7b1da4f0"}
Oct 29 22:45:04.528785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:04.528763    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89" event={"ID":"450a7f81-0bb4-4c1d-9450-13845587b632","Type":"ContainerStarted","Data":"c96d5d627281c35ef3a1be8800f62ce79f5bbb2d259d1f576b4ff0b538fd2b09"}
Oct 29 22:45:04.537991 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:04.533653    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs" event={"ID":"2a0aee5f-3f63-4968-bb93-aec695b43215","Type":"ContainerStarted","Data":"8738c8f8bdbd274e38dc272eedb836e53227f039442b987a78e46565b62caa58"}
Oct 29 22:45:04.541233 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:04.541199    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p" event={"ID":"db5015fd-ba70-49ca-88aa-7f4f657ed546","Type":"ContainerStarted","Data":"19467c49520cb83f71b6dd4cc3562cd2ed2eb39fd6793b109b34e2badb9b2c2a"}
Oct 29 22:45:04.553168 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:04.553140    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:45:04.647659 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:04.647594    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w" podStartSLOduration=4.647575227 podStartE2EDuration="4.647575227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:04.644869619 +0000 UTC m=+1843.556313190" watchObservedRunningTime="2026-10-29 22:45:04.647575227 +0000 UTC m=+1843.559018781"
Oct 29 22:45:04.699836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:04.696010    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg" podStartSLOduration=4.695986181 podStartE2EDuration="4.695986181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:04.685500502 +0000 UTC m=+1843.596944091" watchObservedRunningTime="2026-10-29 22:45:04.695986181 +0000 UTC m=+1843.607429736"
Oct 29 22:45:04.814436 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:04.814364    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2" podStartSLOduration=4.814344802 podStartE2EDuration="4.814344802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:04.810084319 +0000 UTC m=+1843.721527872" watchObservedRunningTime="2026-10-29 22:45:04.814344802 +0000 UTC m=+1843.725788555"
Oct 29 22:45:05.556905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.556764    4559 generic.go:356] "Generic (PLEG): container finished" podID="bb74ed25-e90c-48d8-bfb8-7687f250e690" containerID="a31076e1163e36ce50059c341dde74a7694aa12d340d19bbe7d99fad7b1da4f0" exitCode=60
Oct 29 22:45:05.556905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.556864    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w" event={"ID":"bb74ed25-e90c-48d8-bfb8-7687f250e690","Type":"ContainerDied","Data":"a31076e1163e36ce50059c341dde74a7694aa12d340d19bbe7d99fad7b1da4f0"}
Oct 29 22:45:05.557376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.556938    4559 scope.go:122] "RemoveContainer" containerID="2341a6db2c554a46ef2b96a62e53f7fc26a22b2321ab804bd48e68caca819b00"
Oct 29 22:45:05.557376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.557313    4559 scope.go:122] "RemoveContainer" containerID="a31076e1163e36ce50059c341dde74a7694aa12d340d19bbe7d99fad7b1da4f0"
Oct 29 22:45:05.557797 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:05.557772    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w_odh-ai-gateway-infra(bb74ed25-e90c-48d8-bfb8-7687f250e690)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w" podUID="bb74ed25-e90c-48d8-bfb8-7687f250e690"
Oct 29 22:45:05.559417 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.559389    4559 generic.go:356] "Generic (PLEG): container finished" podID="450a7f81-0bb4-4c1d-9450-13845587b632" containerID="c96d5d627281c35ef3a1be8800f62ce79f5bbb2d259d1f576b4ff0b538fd2b09" exitCode=60
Oct 29 22:45:05.559535 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.559434    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89" event={"ID":"450a7f81-0bb4-4c1d-9450-13845587b632","Type":"ContainerDied","Data":"c96d5d627281c35ef3a1be8800f62ce79f5bbb2d259d1f576b4ff0b538fd2b09"}
Oct 29 22:45:05.559864 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.559845    4559 scope.go:122] "RemoveContainer" containerID="c96d5d627281c35ef3a1be8800f62ce79f5bbb2d259d1f576b4ff0b538fd2b09"
Oct 29 22:45:05.560306 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:05.560284    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89_odh-ai-gateway-infra(450a7f81-0bb4-4c1d-9450-13845587b632)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89" podUID="450a7f81-0bb4-4c1d-9450-13845587b632"
Oct 29 22:45:05.562376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.562351    4559 generic.go:356] "Generic (PLEG): container finished" podID="2a0aee5f-3f63-4968-bb93-aec695b43215" containerID="8738c8f8bdbd274e38dc272eedb836e53227f039442b987a78e46565b62caa58" exitCode=60
Oct 29 22:45:05.562480 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.562457    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs" event={"ID":"2a0aee5f-3f63-4968-bb93-aec695b43215","Type":"ContainerDied","Data":"8738c8f8bdbd274e38dc272eedb836e53227f039442b987a78e46565b62caa58"}
Oct 29 22:45:05.562748 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.562732    4559 scope.go:122] "RemoveContainer" containerID="8738c8f8bdbd274e38dc272eedb836e53227f039442b987a78e46565b62caa58"
Oct 29 22:45:05.563179 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:05.563154    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs_odh-ai-gateway-infra(2a0aee5f-3f63-4968-bb93-aec695b43215)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs" podUID="2a0aee5f-3f63-4968-bb93-aec695b43215"
Oct 29 22:45:05.564937 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.564914    4559 generic.go:356] "Generic (PLEG): container finished" podID="db5015fd-ba70-49ca-88aa-7f4f657ed546" containerID="19467c49520cb83f71b6dd4cc3562cd2ed2eb39fd6793b109b34e2badb9b2c2a" exitCode=60
Oct 29 22:45:05.565043 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.565000    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p" event={"ID":"db5015fd-ba70-49ca-88aa-7f4f657ed546","Type":"ContainerDied","Data":"19467c49520cb83f71b6dd4cc3562cd2ed2eb39fd6793b109b34e2badb9b2c2a"}
Oct 29 22:45:05.565283 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.565262    4559 scope.go:122] "RemoveContainer" containerID="19467c49520cb83f71b6dd4cc3562cd2ed2eb39fd6793b109b34e2badb9b2c2a"
Oct 29 22:45:05.565602 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:05.565585    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888565-gfh9p_odh-ai-gateway-infra(db5015fd-ba70-49ca-88aa-7f4f657ed546)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p" podUID="db5015fd-ba70-49ca-88aa-7f4f657ed546"
Oct 29 22:45:05.567575 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.567548    4559 generic.go:356] "Generic (PLEG): container finished" podID="873fd28e-ffe4-4c7b-80d3-132892b4523e" containerID="43c7c4ccf793186e16d669913abdb3dc256f528990a54d82846871b467382a21" exitCode=60
Oct 29 22:45:05.567687 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.567623    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr" event={"ID":"873fd28e-ffe4-4c7b-80d3-132892b4523e","Type":"ContainerDied","Data":"43c7c4ccf793186e16d669913abdb3dc256f528990a54d82846871b467382a21"}
Oct 29 22:45:05.567938 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.567919    4559 scope.go:122] "RemoveContainer" containerID="43c7c4ccf793186e16d669913abdb3dc256f528990a54d82846871b467382a21"
Oct 29 22:45:05.568328 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:05.568303    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr_odh-ai-gateway-infra(873fd28e-ffe4-4c7b-80d3-132892b4523e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr" podUID="873fd28e-ffe4-4c7b-80d3-132892b4523e"
Oct 29 22:45:05.571351 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.571326    4559 generic.go:356] "Generic (PLEG): container finished" podID="864f2e20-73e9-4552-a8bf-b44acebb7629" containerID="e62ebe3b6bddd6046a7b948912dd41b15259bf2b9f1d589e6234a67a001de1b0" exitCode=60
Oct 29 22:45:05.571503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.571412    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg" event={"ID":"864f2e20-73e9-4552-a8bf-b44acebb7629","Type":"ContainerDied","Data":"e62ebe3b6bddd6046a7b948912dd41b15259bf2b9f1d589e6234a67a001de1b0"}
Oct 29 22:45:05.571899 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.571870    4559 scope.go:122] "RemoveContainer" containerID="e62ebe3b6bddd6046a7b948912dd41b15259bf2b9f1d589e6234a67a001de1b0"
Oct 29 22:45:05.572273 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:05.572249    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg_odh-ai-gateway-infra(864f2e20-73e9-4552-a8bf-b44acebb7629)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg" podUID="864f2e20-73e9-4552-a8bf-b44acebb7629"
Oct 29 22:45:05.575052 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.574986    4559 scope.go:122] "RemoveContainer" containerID="48d27cf7a62aa17d63c88933f123e3e56b8912d2eaabcfa46b88e7a72a1d2ea9"
Oct 29 22:45:05.576889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.576862    4559 generic.go:356] "Generic (PLEG): container finished" podID="698ca9e5-80e2-4475-a865-389166ac7ec9" containerID="dd10e2bc6751650233d89a110cd512a1f0846cdb57bf6788fdb6285ca9ca4e9c" exitCode=60
Oct 29 22:45:05.576990 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.576959    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx" event={"ID":"698ca9e5-80e2-4475-a865-389166ac7ec9","Type":"ContainerDied","Data":"dd10e2bc6751650233d89a110cd512a1f0846cdb57bf6788fdb6285ca9ca4e9c"}
Oct 29 22:45:05.577321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.577300    4559 scope.go:122] "RemoveContainer" containerID="dd10e2bc6751650233d89a110cd512a1f0846cdb57bf6788fdb6285ca9ca4e9c"
Oct 29 22:45:05.577798 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:05.577768    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx_odh-ai-gateway-infra(698ca9e5-80e2-4475-a865-389166ac7ec9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx" podUID="698ca9e5-80e2-4475-a865-389166ac7ec9"
Oct 29 22:45:05.582204 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.582178    4559 scope.go:122] "RemoveContainer" containerID="c0139ffe4206d61c20c334e1debafe7595628a5cdd70ebf4f90ca952797cfaa2"
Oct 29 22:45:05.582508 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:05.582489    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw_odh-ai-gateway-infra(cfaf0d72-b8d6-400c-a285-374310fdace7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw" podUID="cfaf0d72-b8d6-400c-a285-374310fdace7"
Oct 29 22:45:05.585022 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.584998    4559 generic.go:356] "Generic (PLEG): container finished" podID="254d613c-ac01-4d64-90d2-4c4630a1d9f1" containerID="089171882c9dd74091640c95221df23193bbbfdf227986b42f4a4337f4251f63" exitCode=60
Oct 29 22:45:05.585129 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.585081    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2" event={"ID":"254d613c-ac01-4d64-90d2-4c4630a1d9f1","Type":"ContainerDied","Data":"089171882c9dd74091640c95221df23193bbbfdf227986b42f4a4337f4251f63"}
Oct 29 22:45:05.585357 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.585335    4559 scope.go:122] "RemoveContainer" containerID="089171882c9dd74091640c95221df23193bbbfdf227986b42f4a4337f4251f63"
Oct 29 22:45:05.585686 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:05.585664    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2_odh-ai-gateway-infra(254d613c-ac01-4d64-90d2-4c4630a1d9f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2" podUID="254d613c-ac01-4d64-90d2-4c4630a1d9f1"
Oct 29 22:45:05.592006 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.591985    4559 scope.go:122] "RemoveContainer" containerID="1f33c705cb4feb9b0736f534c0370500dac5654f9a4721d6f943658f5181c9e1"
Oct 29 22:45:05.608055 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.608031    4559 scope.go:122] "RemoveContainer" containerID="34bbad1fde0096f8a9b1215842f5c506522f0a362a66da9abe2fe612dc3e798d"
Oct 29 22:45:05.623892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.623870    4559 scope.go:122] "RemoveContainer" containerID="f2ae78db83ec203b386bbc61dd396a293b033c55aa59484d90f8528663917f7e"
Oct 29 22:45:05.640628 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.640600    4559 scope.go:122] "RemoveContainer" containerID="4e01228a8ff9aecb463a84dcb7fb15dc12b7353e05accd0e6437928b61a01dd2"
Oct 29 22:45:05.658017 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.657994    4559 scope.go:122] "RemoveContainer" containerID="27ebd93773f47bab573fdb2842318c4c55a94101abe8aca03a1d7df50d573d37"
Oct 29 22:45:05.674585 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.674560    4559 scope.go:122] "RemoveContainer" containerID="9197e0eec5aa827c894c095afe6732bf9aa976621f8d69e0020401577c344ede"
Oct 29 22:45:05.674907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.674876    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:45:05.692619 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:05.692588    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:45:06.596191 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:06.596163    4559 scope.go:122] "RemoveContainer" containerID="a31076e1163e36ce50059c341dde74a7694aa12d340d19bbe7d99fad7b1da4f0"
Oct 29 22:45:06.596658 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:06.596618    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w_odh-ai-gateway-infra(bb74ed25-e90c-48d8-bfb8-7687f250e690)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w" podUID="bb74ed25-e90c-48d8-bfb8-7687f250e690"
Oct 29 22:45:06.597977 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:06.597953    4559 scope.go:122] "RemoveContainer" containerID="c96d5d627281c35ef3a1be8800f62ce79f5bbb2d259d1f576b4ff0b538fd2b09"
Oct 29 22:45:06.598407 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:06.598369    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89_odh-ai-gateway-infra(450a7f81-0bb4-4c1d-9450-13845587b632)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89" podUID="450a7f81-0bb4-4c1d-9450-13845587b632"
Oct 29 22:45:06.599678 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:06.599658    4559 scope.go:122] "RemoveContainer" containerID="8738c8f8bdbd274e38dc272eedb836e53227f039442b987a78e46565b62caa58"
Oct 29 22:45:06.600080 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:06.600050    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs_odh-ai-gateway-infra(2a0aee5f-3f63-4968-bb93-aec695b43215)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs" podUID="2a0aee5f-3f63-4968-bb93-aec695b43215"
Oct 29 22:45:06.601532 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:06.601510    4559 scope.go:122] "RemoveContainer" containerID="19467c49520cb83f71b6dd4cc3562cd2ed2eb39fd6793b109b34e2badb9b2c2a"
Oct 29 22:45:06.601929 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:06.601908    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888565-gfh9p_odh-ai-gateway-infra(db5015fd-ba70-49ca-88aa-7f4f657ed546)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p" podUID="db5015fd-ba70-49ca-88aa-7f4f657ed546"
Oct 29 22:45:06.603141 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:06.603120    4559 scope.go:122] "RemoveContainer" containerID="43c7c4ccf793186e16d669913abdb3dc256f528990a54d82846871b467382a21"
Oct 29 22:45:06.603486 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:06.603461    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr_odh-ai-gateway-infra(873fd28e-ffe4-4c7b-80d3-132892b4523e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr" podUID="873fd28e-ffe4-4c7b-80d3-132892b4523e"
Oct 29 22:45:06.604739 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:06.604721    4559 scope.go:122] "RemoveContainer" containerID="e62ebe3b6bddd6046a7b948912dd41b15259bf2b9f1d589e6234a67a001de1b0"
Oct 29 22:45:06.605062 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:06.605037    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg_odh-ai-gateway-infra(864f2e20-73e9-4552-a8bf-b44acebb7629)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg" podUID="864f2e20-73e9-4552-a8bf-b44acebb7629"
Oct 29 22:45:06.607291 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:06.607272    4559 scope.go:122] "RemoveContainer" containerID="dd10e2bc6751650233d89a110cd512a1f0846cdb57bf6788fdb6285ca9ca4e9c"
Oct 29 22:45:06.607614 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:06.607593    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx_odh-ai-gateway-infra(698ca9e5-80e2-4475-a865-389166ac7ec9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx" podUID="698ca9e5-80e2-4475-a865-389166ac7ec9"
Oct 29 22:45:06.608892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:06.608872    4559 scope.go:122] "RemoveContainer" containerID="089171882c9dd74091640c95221df23193bbbfdf227986b42f4a4337f4251f63"
Oct 29 22:45:06.609179 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:06.609154    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2_odh-ai-gateway-infra(254d613c-ac01-4d64-90d2-4c4630a1d9f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2" podUID="254d613c-ac01-4d64-90d2-4c4630a1d9f1"
Oct 29 22:45:06.620487 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:06.620467    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:45:07.283637 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:07.283598    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26" podUID="bcd744cd-3522-40f3-9ef9-857fb1ab36e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:45:07.283637 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:07.283616    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k" podUID="b1912dca-8e71-4a6c-ac38-934bd5f4a8df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:45:08.789587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:08.789552    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:45:08.802107 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:08.802078    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:45:10.294103 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:10.294069    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:45:13.798869 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:13.798822    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:45:15.220990 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:15.220953    4559 scope.go:122] "RemoveContainer" containerID="e62ebe3b6bddd6046a7b948912dd41b15259bf2b9f1d589e6234a67a001de1b0"
Oct 29 22:45:16.220652 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:16.220607    4559 scope.go:122] "RemoveContainer" containerID="089171882c9dd74091640c95221df23193bbbfdf227986b42f4a4337f4251f63"
Oct 29 22:45:16.220971 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:16.220942    4559 scope.go:122] "RemoveContainer" containerID="19467c49520cb83f71b6dd4cc3562cd2ed2eb39fd6793b109b34e2badb9b2c2a"
Oct 29 22:45:16.221057 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:16.220990    4559 scope.go:122] "RemoveContainer" containerID="c0139ffe4206d61c20c334e1debafe7595628a5cdd70ebf4f90ca952797cfaa2"
Oct 29 22:45:16.221437 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:16.221316    4559 scope.go:122] "RemoveContainer" containerID="c96d5d627281c35ef3a1be8800f62ce79f5bbb2d259d1f576b4ff0b538fd2b09"
Oct 29 22:45:16.767766 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:16.767727    4559 generic.go:356] "Generic (PLEG): container finished" podID="864f2e20-73e9-4552-a8bf-b44acebb7629" containerID="2b797cfcf6efccf3d3ceacae0bfe3da4de76298710d8c4bdfa30bb11f52ee4a6" exitCode=60
Oct 29 22:45:16.768007 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:16.767791    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg" event={"ID":"864f2e20-73e9-4552-a8bf-b44acebb7629","Type":"ContainerDied","Data":"2b797cfcf6efccf3d3ceacae0bfe3da4de76298710d8c4bdfa30bb11f52ee4a6"}
Oct 29 22:45:16.768007 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:16.767885    4559 scope.go:122] "RemoveContainer" containerID="e62ebe3b6bddd6046a7b948912dd41b15259bf2b9f1d589e6234a67a001de1b0"
Oct 29 22:45:16.768394 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:16.768376    4559 scope.go:122] "RemoveContainer" containerID="2b797cfcf6efccf3d3ceacae0bfe3da4de76298710d8c4bdfa30bb11f52ee4a6"
Oct 29 22:45:16.768850 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:16.768804    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg_odh-ai-gateway-infra(864f2e20-73e9-4552-a8bf-b44acebb7629)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg" podUID="864f2e20-73e9-4552-a8bf-b44acebb7629"
Oct 29 22:45:17.094631 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.094597    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:45:17.221071 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.221040    4559 scope.go:122] "RemoveContainer" containerID="dd10e2bc6751650233d89a110cd512a1f0846cdb57bf6788fdb6285ca9ca4e9c"
Oct 29 22:45:17.221475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.221204    4559 scope.go:122] "RemoveContainer" containerID="43c7c4ccf793186e16d669913abdb3dc256f528990a54d82846871b467382a21"
Oct 29 22:45:17.284105 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.284068    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26" podUID="bcd744cd-3522-40f3-9ef9-857fb1ab36e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:45:17.284297 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.284205    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k" podUID="b1912dca-8e71-4a6c-ac38-934bd5f4a8df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:45:17.780385 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.780354    4559 generic.go:356] "Generic (PLEG): container finished" podID="cfaf0d72-b8d6-400c-a285-374310fdace7" containerID="32529d7af8ba0f78c08f8c04cd000b40c51ffa14e5d793eb1fa56fd761196d14" exitCode=60
Oct 29 22:45:17.780586 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.780422    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw" event={"ID":"cfaf0d72-b8d6-400c-a285-374310fdace7","Type":"ContainerDied","Data":"32529d7af8ba0f78c08f8c04cd000b40c51ffa14e5d793eb1fa56fd761196d14"}
Oct 29 22:45:17.780586 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.780515    4559 scope.go:122] "RemoveContainer" containerID="c0139ffe4206d61c20c334e1debafe7595628a5cdd70ebf4f90ca952797cfaa2"
Oct 29 22:45:17.781359 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.781317    4559 scope.go:122] "RemoveContainer" containerID="32529d7af8ba0f78c08f8c04cd000b40c51ffa14e5d793eb1fa56fd761196d14"
Oct 29 22:45:17.781854 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:17.781791    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw_odh-ai-gateway-infra(cfaf0d72-b8d6-400c-a285-374310fdace7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw" podUID="cfaf0d72-b8d6-400c-a285-374310fdace7"
Oct 29 22:45:17.784789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.784764    4559 generic.go:356] "Generic (PLEG): container finished" podID="254d613c-ac01-4d64-90d2-4c4630a1d9f1" containerID="9432453529b7313ff8b03010c425177a6b220c5df712b0025cf8affc20ae1588" exitCode=60
Oct 29 22:45:17.784935 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.784835    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2" event={"ID":"254d613c-ac01-4d64-90d2-4c4630a1d9f1","Type":"ContainerDied","Data":"9432453529b7313ff8b03010c425177a6b220c5df712b0025cf8affc20ae1588"}
Oct 29 22:45:17.785280 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.785259    4559 scope.go:122] "RemoveContainer" containerID="9432453529b7313ff8b03010c425177a6b220c5df712b0025cf8affc20ae1588"
Oct 29 22:45:17.785703 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:17.785671    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2_odh-ai-gateway-infra(254d613c-ac01-4d64-90d2-4c4630a1d9f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2" podUID="254d613c-ac01-4d64-90d2-4c4630a1d9f1"
Oct 29 22:45:17.787651 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.787629    4559 generic.go:356] "Generic (PLEG): container finished" podID="450a7f81-0bb4-4c1d-9450-13845587b632" containerID="16e8c94fafa3013b11270295e75c84bdb0f469f86660b2fa0bdadfd3ca585084" exitCode=60
Oct 29 22:45:17.787754 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.787698    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89" event={"ID":"450a7f81-0bb4-4c1d-9450-13845587b632","Type":"ContainerDied","Data":"16e8c94fafa3013b11270295e75c84bdb0f469f86660b2fa0bdadfd3ca585084"}
Oct 29 22:45:17.788165 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.788148    4559 scope.go:122] "RemoveContainer" containerID="16e8c94fafa3013b11270295e75c84bdb0f469f86660b2fa0bdadfd3ca585084"
Oct 29 22:45:17.788572 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:17.788549    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89_odh-ai-gateway-infra(450a7f81-0bb4-4c1d-9450-13845587b632)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89" podUID="450a7f81-0bb4-4c1d-9450-13845587b632"
Oct 29 22:45:17.790141 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.790122    4559 generic.go:356] "Generic (PLEG): container finished" podID="db5015fd-ba70-49ca-88aa-7f4f657ed546" containerID="f5036d8ff2794fa195c9568099b1463bba523fff3f9cc089a78bd961bf5020af" exitCode=60
Oct 29 22:45:17.790247 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.790140    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p" event={"ID":"db5015fd-ba70-49ca-88aa-7f4f657ed546","Type":"ContainerDied","Data":"f5036d8ff2794fa195c9568099b1463bba523fff3f9cc089a78bd961bf5020af"}
Oct 29 22:45:17.790505 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.790489    4559 scope.go:122] "RemoveContainer" containerID="f5036d8ff2794fa195c9568099b1463bba523fff3f9cc089a78bd961bf5020af"
Oct 29 22:45:17.790892 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:17.790870    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888565-gfh9p_odh-ai-gateway-infra(db5015fd-ba70-49ca-88aa-7f4f657ed546)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p" podUID="db5015fd-ba70-49ca-88aa-7f4f657ed546"
Oct 29 22:45:17.793197 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.793172    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr" event={"ID":"873fd28e-ffe4-4c7b-80d3-132892b4523e","Type":"ContainerStarted","Data":"9022f1cff8a23951926b6ec40030ab9d1074d14f60933d8a6e05c5545c62ff38"}
Oct 29 22:45:17.797090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.797032    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx" event={"ID":"698ca9e5-80e2-4475-a865-389166ac7ec9","Type":"ContainerStarted","Data":"69558278c377c6931792cfbc59ed7c5d0c2455698887f6cafaed9eaef73621a8"}
Oct 29 22:45:17.832672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.832646    4559 scope.go:122] "RemoveContainer" containerID="089171882c9dd74091640c95221df23193bbbfdf227986b42f4a4337f4251f63"
Oct 29 22:45:17.853395 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.832797    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg"]
Oct 29 22:45:17.876417 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.876382    4559 scope.go:122] "RemoveContainer" containerID="c96d5d627281c35ef3a1be8800f62ce79f5bbb2d259d1f576b4ff0b538fd2b09"
Oct 29 22:45:17.905649 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:17.905618    4559 scope.go:122] "RemoveContainer" containerID="19467c49520cb83f71b6dd4cc3562cd2ed2eb39fd6793b109b34e2badb9b2c2a"
Oct 29 22:45:18.078541 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:18.078515    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg"
Oct 29 22:45:18.162517 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:18.162481    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/864f2e20-73e9-4552-a8bf-b44acebb7629-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/864f2e20-73e9-4552-a8bf-b44acebb7629-maas-api-service-ca\") pod \"864f2e20-73e9-4552-a8bf-b44acebb7629\" (UID: \"864f2e20-73e9-4552-a8bf-b44acebb7629\") "
Oct 29 22:45:18.162892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:18.162861    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864f2e20-73e9-4552-a8bf-b44acebb7629-maas-api-service-ca" pod "864f2e20-73e9-4552-a8bf-b44acebb7629" (UID: "864f2e20-73e9-4552-a8bf-b44acebb7629"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:18.167539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:18.165504    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864f2e20-73e9-4552-a8bf-b44acebb7629-kube-api-access-r59qf\" (UniqueName: \"kubernetes.io/projected/864f2e20-73e9-4552-a8bf-b44acebb7629-kube-api-access-r59qf\") pod \"864f2e20-73e9-4552-a8bf-b44acebb7629\" (UID: \"864f2e20-73e9-4552-a8bf-b44acebb7629\") "
Oct 29 22:45:18.167539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:18.166367    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/864f2e20-73e9-4552-a8bf-b44acebb7629-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:18.169530 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:18.169495    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864f2e20-73e9-4552-a8bf-b44acebb7629-kube-api-access-r59qf" pod "864f2e20-73e9-4552-a8bf-b44acebb7629" (UID: "864f2e20-73e9-4552-a8bf-b44acebb7629"). InnerVolumeSpecName "kube-api-access-r59qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:18.268344 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:18.268302    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r59qf\" (UniqueName: \"kubernetes.io/projected/864f2e20-73e9-4552-a8bf-b44acebb7629-kube-api-access-r59qf\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:18.878080 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:18.878033    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw"]
Oct 29 22:45:18.989278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:18.987435    4559 generic.go:356] "Generic (PLEG): container finished" podID="873fd28e-ffe4-4c7b-80d3-132892b4523e" containerID="9022f1cff8a23951926b6ec40030ab9d1074d14f60933d8a6e05c5545c62ff38" exitCode=60
Oct 29 22:45:18.989278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:18.987538    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr" event={"ID":"873fd28e-ffe4-4c7b-80d3-132892b4523e","Type":"ContainerDied","Data":"9022f1cff8a23951926b6ec40030ab9d1074d14f60933d8a6e05c5545c62ff38"}
Oct 29 22:45:18.989278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:18.987601    4559 scope.go:122] "RemoveContainer" containerID="43c7c4ccf793186e16d669913abdb3dc256f528990a54d82846871b467382a21"
Oct 29 22:45:18.989278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:18.988143    4559 scope.go:122] "RemoveContainer" containerID="9022f1cff8a23951926b6ec40030ab9d1074d14f60933d8a6e05c5545c62ff38"
Oct 29 22:45:18.989278 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:18.988675    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr_odh-ai-gateway-infra(873fd28e-ffe4-4c7b-80d3-132892b4523e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr" podUID="873fd28e-ffe4-4c7b-80d3-132892b4523e"
Oct 29 22:45:19.025001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.007337    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg" event={"ID":"864f2e20-73e9-4552-a8bf-b44acebb7629","Type":"ContainerDied","Data":"f235e07d484f819a178c5bb686dc6241526bed87b6fc27ba10eb972b0e59b3de"}
Oct 29 22:45:19.025001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.007450    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg"
Oct 29 22:45:19.114105 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.114066    4559 generic.go:356] "Generic (PLEG): container finished" podID="698ca9e5-80e2-4475-a865-389166ac7ec9" containerID="69558278c377c6931792cfbc59ed7c5d0c2455698887f6cafaed9eaef73621a8" exitCode=60
Oct 29 22:45:19.114574 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.114340    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx" event={"ID":"698ca9e5-80e2-4475-a865-389166ac7ec9","Type":"ContainerDied","Data":"69558278c377c6931792cfbc59ed7c5d0c2455698887f6cafaed9eaef73621a8"}
Oct 29 22:45:19.173740 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.173713    4559 scope.go:122] "RemoveContainer" containerID="69558278c377c6931792cfbc59ed7c5d0c2455698887f6cafaed9eaef73621a8"
Oct 29 22:45:19.174525 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:19.174473    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx_odh-ai-gateway-infra(698ca9e5-80e2-4475-a865-389166ac7ec9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx" podUID="698ca9e5-80e2-4475-a865-389166ac7ec9"
Oct 29 22:45:19.198045 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.186305    4559 scope.go:122] "RemoveContainer" containerID="2b797cfcf6efccf3d3ceacae0bfe3da4de76298710d8c4bdfa30bb11f52ee4a6"
Oct 29 22:45:19.470951 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.392433    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4" podUID="e44215a6-17c6-48e9-b4d6-123b1ef1ac12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.144:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:19.524914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.522954    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs" podUID="62156571-1397-4073-ab41-ed9b8038f06a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:19.560482 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.558221    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f" podUID="ea5e69a0-9148-479e-8590-48f3a4848d5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.164:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:19.590882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.590615    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:19.626916 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.613499    4559 scope.go:122] "RemoveContainer" containerID="a31076e1163e36ce50059c341dde74a7694aa12d340d19bbe7d99fad7b1da4f0"
Oct 29 22:45:19.626916 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.621930    4559 scope.go:122] "RemoveContainer" containerID="8738c8f8bdbd274e38dc272eedb836e53227f039442b987a78e46565b62caa58"
Oct 29 22:45:19.626916 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.623648    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz" podUID="568eefef-fcf7-445b-8dfd-a6b7e79215ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.165:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:19.666109 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.666075    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:19.670489 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.670426    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:19.683948 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.683905    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd" podUID="733bd7c3-3efb-4004-bdd9-10e01133855c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.166:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:19.735289 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.721562    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4" podUID="e44215a6-17c6-48e9-b4d6-123b1ef1ac12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.144:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:19.846380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.818079    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd" podUID="733bd7c3-3efb-4004-bdd9-10e01133855c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.166:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:19.997893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:19.987503    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr"]
Oct 29 22:45:20.117992 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:20.117946    4559 scope.go:122] "RemoveContainer" containerID="dd10e2bc6751650233d89a110cd512a1f0846cdb57bf6788fdb6285ca9ca4e9c"
Oct 29 22:45:20.252894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:20.242889    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx"]
Oct 29 22:45:20.568996 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:20.568896    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p"]
Oct 29 22:45:20.623125 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:20.608040    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:20.730843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:20.648008    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz" podUID="568eefef-fcf7-445b-8dfd-a6b7e79215ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.165:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:20.730843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:20.649455    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:45:20.752543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:20.731454    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd" podUID="733bd7c3-3efb-4004-bdd9-10e01133855c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.166:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:20.752543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:20.751335    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:45:21.311895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.311833    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:45:21.591435 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.591381    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:21.771900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.771855    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd" podUID="733bd7c3-3efb-4004-bdd9-10e01133855c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.166:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:21.841775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.834171    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:45:21.841775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.834214    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2"]
Oct 29 22:45:21.841775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.834275    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:45:21.856513 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.856469    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89"]
Oct 29 22:45:21.872831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.872710    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"]
Oct 29 22:45:21.873446 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.873090    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm" podUID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" containerName="istio-proxy" containerID="cri-o://9dd62e009a5921088740e5c1727fd5a7cb015c0b386742ecaf8e17ceb7c4646f" gracePeriod=30
Oct 29 22:45:21.877994 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.877886    4559 generic.go:356] "Generic (PLEG): container finished" podID="bcd744cd-3522-40f3-9ef9-857fb1ab36e9" containerID="dc102aff2c38a8f7790d22e14c4cbca22851ab2d711650fbb3c72972bcb75b91" exitCode=0
Oct 29 22:45:21.878231 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.878183    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26" event={"ID":"bcd744cd-3522-40f3-9ef9-857fb1ab36e9","Type":"ContainerDied","Data":"dc102aff2c38a8f7790d22e14c4cbca22851ab2d711650fbb3c72972bcb75b91"}
Oct 29 22:45:21.883889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.883862    4559 generic.go:356] "Generic (PLEG): container finished" podID="b1912dca-8e71-4a6c-ac38-934bd5f4a8df" containerID="d5d86c226fb1102a3cefe2bdcb8751ce4d6c2a4cbdb7ffc7165ed7372e0c260b" exitCode=0
Oct 29 22:45:21.885187 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.885151    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k" event={"ID":"b1912dca-8e71-4a6c-ac38-934bd5f4a8df","Type":"ContainerDied","Data":"d5d86c226fb1102a3cefe2bdcb8751ce4d6c2a4cbdb7ffc7165ed7372e0c260b"}
Oct 29 22:45:21.913471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.907110    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"]
Oct 29 22:45:21.913471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.908695    4559 cpu_manager.go:414] "containerMap: removing container" podUID="864f2e20-73e9-4552-a8bf-b44acebb7629" containerName="cleanup"
Oct 29 22:45:21.913471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.908721    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864f2e20-73e9-4552-a8bf-b44acebb7629" containerName="cleanup"
Oct 29 22:45:21.913471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.908746    4559 cpu_manager.go:414] "containerMap: removing container" podUID="864f2e20-73e9-4552-a8bf-b44acebb7629" containerName="cleanup"
Oct 29 22:45:21.913471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.908754    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864f2e20-73e9-4552-a8bf-b44acebb7629" containerName="cleanup"
Oct 29 22:45:21.913471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.908764    4559 cpu_manager.go:414] "containerMap: removing container" podUID="864f2e20-73e9-4552-a8bf-b44acebb7629" containerName="cleanup"
Oct 29 22:45:21.913471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.908771    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864f2e20-73e9-4552-a8bf-b44acebb7629" containerName="cleanup"
Oct 29 22:45:21.913471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.909104    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="864f2e20-73e9-4552-a8bf-b44acebb7629" containerName="cleanup"
Oct 29 22:45:21.913471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.909120    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="864f2e20-73e9-4552-a8bf-b44acebb7629" containerName="cleanup"
Oct 29 22:45:21.918741 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.914489    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw"
Oct 29 22:45:21.931108 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.931076    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg"]
Oct 29 22:45:21.931323 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.931308    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888565-svjpg"]
Oct 29 22:45:21.931470 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.931458    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"]
Oct 29 22:45:21.931703 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.931687    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"
Oct 29 22:45:21.936355 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.935219    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6324171d-890b-481b-bf15-29dec862d313-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz\" (UID: \"6324171d-890b-481b-bf15-29dec862d313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"
Oct 29 22:45:21.936355 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.935253    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6324171d-890b-481b-bf15-29dec862d313-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz\" (UID: \"6324171d-890b-481b-bf15-29dec862d313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"
Oct 29 22:45:21.936355 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.935468    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfw48\" (UniqueName: \"kubernetes.io/projected/6324171d-890b-481b-bf15-29dec862d313-kube-api-access-sfw48\") pod \"maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz\" (UID: \"6324171d-890b-481b-bf15-29dec862d313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"
Oct 29 22:45:21.947600 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.947539    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:45:21.948907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.948459    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"
Oct 29 22:45:21.952296 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.951096    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"
Oct 29 22:45:21.958690 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:21.958607    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p"
Oct 29 22:45:22.041832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.038389    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfaf0d72-b8d6-400c-a285-374310fdace7-kube-api-access-ws5ns\" (UniqueName: \"kubernetes.io/projected/cfaf0d72-b8d6-400c-a285-374310fdace7-kube-api-access-ws5ns\") pod \"cfaf0d72-b8d6-400c-a285-374310fdace7\" (UID: \"cfaf0d72-b8d6-400c-a285-374310fdace7\") "
Oct 29 22:45:22.041832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.038580    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfaf0d72-b8d6-400c-a285-374310fdace7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaf0d72-b8d6-400c-a285-374310fdace7-maas-api-service-ca\") pod \"cfaf0d72-b8d6-400c-a285-374310fdace7\" (UID: \"cfaf0d72-b8d6-400c-a285-374310fdace7\") "
Oct 29 22:45:22.041832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.039330    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfw48\" (UniqueName: \"kubernetes.io/projected/6324171d-890b-481b-bf15-29dec862d313-kube-api-access-sfw48\") pod \"maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz\" (UID: \"6324171d-890b-481b-bf15-29dec862d313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"
Oct 29 22:45:22.041832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.039636    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6324171d-890b-481b-bf15-29dec862d313-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz\" (UID: \"6324171d-890b-481b-bf15-29dec862d313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"
Oct 29 22:45:22.041832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.039668    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6324171d-890b-481b-bf15-29dec862d313-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz\" (UID: \"6324171d-890b-481b-bf15-29dec862d313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"
Oct 29 22:45:22.041832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.041293    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6324171d-890b-481b-bf15-29dec862d313-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz\" (UID: \"6324171d-890b-481b-bf15-29dec862d313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"
Oct 29 22:45:22.041832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.041782    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfaf0d72-b8d6-400c-a285-374310fdace7-maas-api-service-ca" pod "cfaf0d72-b8d6-400c-a285-374310fdace7" (UID: "cfaf0d72-b8d6-400c-a285-374310fdace7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:22.048553 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.047605    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6324171d-890b-481b-bf15-29dec862d313-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz\" (UID: \"6324171d-890b-481b-bf15-29dec862d313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"
Oct 29 22:45:22.049283 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.049245    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfaf0d72-b8d6-400c-a285-374310fdace7-kube-api-access-ws5ns" pod "cfaf0d72-b8d6-400c-a285-374310fdace7" (UID: "cfaf0d72-b8d6-400c-a285-374310fdace7"). InnerVolumeSpecName "kube-api-access-ws5ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:22.068700 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.068588    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfw48\" (UniqueName: \"kubernetes.io/projected/6324171d-890b-481b-bf15-29dec862d313-kube-api-access-sfw48\") pod \"maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz\" (UID: \"6324171d-890b-481b-bf15-29dec862d313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"
Oct 29 22:45:22.101622 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.101456    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr"
Oct 29 22:45:22.125146 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.125110    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2"
Oct 29 22:45:22.140447 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.140414    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1912dca-8e71-4a6c-ac38-934bd5f4a8df-kube-api-access-pdd9l\" (UniqueName: \"kubernetes.io/projected/b1912dca-8e71-4a6c-ac38-934bd5f4a8df-kube-api-access-pdd9l\") pod \"b1912dca-8e71-4a6c-ac38-934bd5f4a8df\" (UID: \"b1912dca-8e71-4a6c-ac38-934bd5f4a8df\") "
Oct 29 22:45:22.140652 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.140578    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1912dca-8e71-4a6c-ac38-934bd5f4a8df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1912dca-8e71-4a6c-ac38-934bd5f4a8df-plugins-config-volume\") pod \"b1912dca-8e71-4a6c-ac38-934bd5f4a8df\" (UID: \"b1912dca-8e71-4a6c-ac38-934bd5f4a8df\") "
Oct 29 22:45:22.140652 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.140603    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5015fd-ba70-49ca-88aa-7f4f657ed546-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5015fd-ba70-49ca-88aa-7f4f657ed546-maas-api-service-ca\") pod \"db5015fd-ba70-49ca-88aa-7f4f657ed546\" (UID: \"db5015fd-ba70-49ca-88aa-7f4f657ed546\") "
Oct 29 22:45:22.140794 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.140670    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd744cd-3522-40f3-9ef9-857fb1ab36e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcd744cd-3522-40f3-9ef9-857fb1ab36e9-plugins-config-volume\") pod \"bcd744cd-3522-40f3-9ef9-857fb1ab36e9\" (UID: \"bcd744cd-3522-40f3-9ef9-857fb1ab36e9\") "
Oct 29 22:45:22.140875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.140796    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db5015fd-ba70-49ca-88aa-7f4f657ed546-kube-api-access-nmlww\" (UniqueName: \"kubernetes.io/projected/db5015fd-ba70-49ca-88aa-7f4f657ed546-kube-api-access-nmlww\") pod \"db5015fd-ba70-49ca-88aa-7f4f657ed546\" (UID: \"db5015fd-ba70-49ca-88aa-7f4f657ed546\") "
Oct 29 22:45:22.140875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.140867    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd744cd-3522-40f3-9ef9-857fb1ab36e9-kube-api-access-sbmk7\" (UniqueName: \"kubernetes.io/projected/bcd744cd-3522-40f3-9ef9-857fb1ab36e9-kube-api-access-sbmk7\") pod \"bcd744cd-3522-40f3-9ef9-857fb1ab36e9\" (UID: \"bcd744cd-3522-40f3-9ef9-857fb1ab36e9\") "
Oct 29 22:45:22.141447 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.141424    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaf0d72-b8d6-400c-a285-374310fdace7-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.141447 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.141446    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws5ns\" (UniqueName: \"kubernetes.io/projected/cfaf0d72-b8d6-400c-a285-374310fdace7-kube-api-access-ws5ns\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.142482 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.142431    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5015fd-ba70-49ca-88aa-7f4f657ed546-maas-api-service-ca" pod "db5015fd-ba70-49ca-88aa-7f4f657ed546" (UID: "db5015fd-ba70-49ca-88aa-7f4f657ed546"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:22.142607 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.142459    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd744cd-3522-40f3-9ef9-857fb1ab36e9-plugins-config-volume" pod "bcd744cd-3522-40f3-9ef9-857fb1ab36e9" (UID: "bcd744cd-3522-40f3-9ef9-857fb1ab36e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:22.143167 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.143146    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1912dca-8e71-4a6c-ac38-934bd5f4a8df-plugins-config-volume" pod "b1912dca-8e71-4a6c-ac38-934bd5f4a8df" (UID: "b1912dca-8e71-4a6c-ac38-934bd5f4a8df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:22.144944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.144736    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5015fd-ba70-49ca-88aa-7f4f657ed546-kube-api-access-nmlww" pod "db5015fd-ba70-49ca-88aa-7f4f657ed546" (UID: "db5015fd-ba70-49ca-88aa-7f4f657ed546"). InnerVolumeSpecName "kube-api-access-nmlww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:22.145520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.145493    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd744cd-3522-40f3-9ef9-857fb1ab36e9-kube-api-access-sbmk7" pod "bcd744cd-3522-40f3-9ef9-857fb1ab36e9" (UID: "bcd744cd-3522-40f3-9ef9-857fb1ab36e9"). InnerVolumeSpecName "kube-api-access-sbmk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:22.147124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.147098    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1912dca-8e71-4a6c-ac38-934bd5f4a8df-kube-api-access-pdd9l" pod "b1912dca-8e71-4a6c-ac38-934bd5f4a8df" (UID: "b1912dca-8e71-4a6c-ac38-934bd5f4a8df"). InnerVolumeSpecName "kube-api-access-pdd9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:22.228903 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.228773    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89"
Oct 29 22:45:22.244135 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.243347    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/873fd28e-ffe4-4c7b-80d3-132892b4523e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/873fd28e-ffe4-4c7b-80d3-132892b4523e-maas-api-service-ca\") pod \"873fd28e-ffe4-4c7b-80d3-132892b4523e\" (UID: \"873fd28e-ffe4-4c7b-80d3-132892b4523e\") "
Oct 29 22:45:22.244135 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.243415    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254d613c-ac01-4d64-90d2-4c4630a1d9f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254d613c-ac01-4d64-90d2-4c4630a1d9f1-maas-api-service-ca\") pod \"254d613c-ac01-4d64-90d2-4c4630a1d9f1\" (UID: \"254d613c-ac01-4d64-90d2-4c4630a1d9f1\") "
Oct 29 22:45:22.244135 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.243548    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/873fd28e-ffe4-4c7b-80d3-132892b4523e-kube-api-access-68q7q\" (UniqueName: \"kubernetes.io/projected/873fd28e-ffe4-4c7b-80d3-132892b4523e-kube-api-access-68q7q\") pod \"873fd28e-ffe4-4c7b-80d3-132892b4523e\" (UID: \"873fd28e-ffe4-4c7b-80d3-132892b4523e\") "
Oct 29 22:45:22.244135 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.243580    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254d613c-ac01-4d64-90d2-4c4630a1d9f1-kube-api-access-zcvvj\" (UniqueName: \"kubernetes.io/projected/254d613c-ac01-4d64-90d2-4c4630a1d9f1-kube-api-access-zcvvj\") pod \"254d613c-ac01-4d64-90d2-4c4630a1d9f1\" (UID: \"254d613c-ac01-4d64-90d2-4c4630a1d9f1\") "
Oct 29 22:45:22.244522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.244355    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/873fd28e-ffe4-4c7b-80d3-132892b4523e-maas-api-service-ca" pod "873fd28e-ffe4-4c7b-80d3-132892b4523e" (UID: "873fd28e-ffe4-4c7b-80d3-132892b4523e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:22.244522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.244379    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1912dca-8e71-4a6c-ac38-934bd5f4a8df-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.244522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.244407    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5015fd-ba70-49ca-88aa-7f4f657ed546-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.244522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.244422    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcd744cd-3522-40f3-9ef9-857fb1ab36e9-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.244522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.244439    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmlww\" (UniqueName: \"kubernetes.io/projected/db5015fd-ba70-49ca-88aa-7f4f657ed546-kube-api-access-nmlww\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.244522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.244453    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbmk7\" (UniqueName: \"kubernetes.io/projected/bcd744cd-3522-40f3-9ef9-857fb1ab36e9-kube-api-access-sbmk7\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.244522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.244468    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdd9l\" (UniqueName: \"kubernetes.io/projected/b1912dca-8e71-4a6c-ac38-934bd5f4a8df-kube-api-access-pdd9l\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.247926 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.247891    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254d613c-ac01-4d64-90d2-4c4630a1d9f1-kube-api-access-zcvvj" pod "254d613c-ac01-4d64-90d2-4c4630a1d9f1" (UID: "254d613c-ac01-4d64-90d2-4c4630a1d9f1"). InnerVolumeSpecName "kube-api-access-zcvvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:22.250968 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.249909    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873fd28e-ffe4-4c7b-80d3-132892b4523e-kube-api-access-68q7q" pod "873fd28e-ffe4-4c7b-80d3-132892b4523e" (UID: "873fd28e-ffe4-4c7b-80d3-132892b4523e"). InnerVolumeSpecName "kube-api-access-68q7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:22.250968 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.250228    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254d613c-ac01-4d64-90d2-4c4630a1d9f1-maas-api-service-ca" pod "254d613c-ac01-4d64-90d2-4c4630a1d9f1" (UID: "254d613c-ac01-4d64-90d2-4c4630a1d9f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:22.261352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.261323    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx"
Oct 29 22:45:22.278512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.278477    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"
Oct 29 22:45:22.346219 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.346192    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450a7f81-0bb4-4c1d-9450-13845587b632-kube-api-access-89dxf\" (UniqueName: \"kubernetes.io/projected/450a7f81-0bb4-4c1d-9450-13845587b632-kube-api-access-89dxf\") pod \"450a7f81-0bb4-4c1d-9450-13845587b632\" (UID: \"450a7f81-0bb4-4c1d-9450-13845587b632\") "
Oct 29 22:45:22.346621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.346525    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/450a7f81-0bb4-4c1d-9450-13845587b632-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/450a7f81-0bb4-4c1d-9450-13845587b632-maas-api-service-ca\") pod \"450a7f81-0bb4-4c1d-9450-13845587b632\" (UID: \"450a7f81-0bb4-4c1d-9450-13845587b632\") "
Oct 29 22:45:22.347672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.347644    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/873fd28e-ffe4-4c7b-80d3-132892b4523e-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.347800 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.347696    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254d613c-ac01-4d64-90d2-4c4630a1d9f1-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.347800 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.347718    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68q7q\" (UniqueName: \"kubernetes.io/projected/873fd28e-ffe4-4c7b-80d3-132892b4523e-kube-api-access-68q7q\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.347800 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.347744    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcvvj\" (UniqueName: \"kubernetes.io/projected/254d613c-ac01-4d64-90d2-4c4630a1d9f1-kube-api-access-zcvvj\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.350176 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.350120    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450a7f81-0bb4-4c1d-9450-13845587b632-maas-api-service-ca" pod "450a7f81-0bb4-4c1d-9450-13845587b632" (UID: "450a7f81-0bb4-4c1d-9450-13845587b632"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:22.364606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.364564    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450a7f81-0bb4-4c1d-9450-13845587b632-kube-api-access-89dxf" pod "450a7f81-0bb4-4c1d-9450-13845587b632" (UID: "450a7f81-0bb4-4c1d-9450-13845587b632"). InnerVolumeSpecName "kube-api-access-89dxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:22.466916 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.456589    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698ca9e5-80e2-4475-a865-389166ac7ec9-kube-api-access-zwzh2\" (UniqueName: \"kubernetes.io/projected/698ca9e5-80e2-4475-a865-389166ac7ec9-kube-api-access-zwzh2\") pod \"698ca9e5-80e2-4475-a865-389166ac7ec9\" (UID: \"698ca9e5-80e2-4475-a865-389166ac7ec9\") "
Oct 29 22:45:22.467319 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.467293    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/698ca9e5-80e2-4475-a865-389166ac7ec9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/698ca9e5-80e2-4475-a865-389166ac7ec9-maas-api-service-ca\") pod \"698ca9e5-80e2-4475-a865-389166ac7ec9\" (UID: \"698ca9e5-80e2-4475-a865-389166ac7ec9\") "
Oct 29 22:45:22.468451 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.468410    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/698ca9e5-80e2-4475-a865-389166ac7ec9-maas-api-service-ca" pod "698ca9e5-80e2-4475-a865-389166ac7ec9" (UID: "698ca9e5-80e2-4475-a865-389166ac7ec9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:22.469856 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.469829    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89dxf\" (UniqueName: \"kubernetes.io/projected/450a7f81-0bb4-4c1d-9450-13845587b632-kube-api-access-89dxf\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.469999 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.469986    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/450a7f81-0bb4-4c1d-9450-13845587b632-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.470087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.470076    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/698ca9e5-80e2-4475-a865-389166ac7ec9-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.478070 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.478029    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698ca9e5-80e2-4475-a865-389166ac7ec9-kube-api-access-zwzh2" pod "698ca9e5-80e2-4475-a865-389166ac7ec9" (UID: "698ca9e5-80e2-4475-a865-389166ac7ec9"). InnerVolumeSpecName "kube-api-access-zwzh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:22.575495 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.575448    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwzh2\" (UniqueName: \"kubernetes.io/projected/698ca9e5-80e2-4475-a865-389166ac7ec9-kube-api-access-zwzh2\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.643327 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.643299    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"]
Oct 29 22:45:22.646732 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:45:22.646700    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6324171d_890b_481b_bf15_29dec862d313.slice/crio-95678cdadfbc590887284f665318bd5fef51d9cad8db34fff09e6006d0610fa3 WatchSource:0}: Error finding container 95678cdadfbc590887284f665318bd5fef51d9cad8db34fff09e6006d0610fa3: Status 404 returned error can't find the container with id 95678cdadfbc590887284f665318bd5fef51d9cad8db34fff09e6006d0610fa3
Oct 29 22:45:22.799632 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.799593    4559 scope.go:122] "RemoveContainer" containerID="d5d86c226fb1102a3cefe2bdcb8751ce4d6c2a4cbdb7ffc7165ed7372e0c260b"
Oct 29 22:45:22.812232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.812203    4559 scope.go:122] "RemoveContainer" containerID="dc102aff2c38a8f7790d22e14c4cbca22851ab2d711650fbb3c72972bcb75b91"
Oct 29 22:45:22.824127 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.824085    4559 scope.go:122] "RemoveContainer" containerID="deac0201b3c242632a856d6ce23e0a8b86b0a9b2d5bdb8efdbb4398c65886945"
Oct 29 22:45:22.898539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.898508    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89" event={"ID":"450a7f81-0bb4-4c1d-9450-13845587b632","Type":"ContainerDied","Data":"bee9f9d11f93ffae1893ebff2ab50c3b1e618abe8aaa4de847c0c72a3a0e3f0a"}
Oct 29 22:45:22.898539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.898529    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89"
Oct 29 22:45:22.898682 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.898619    4559 scope.go:122] "RemoveContainer" containerID="16e8c94fafa3013b11270295e75c84bdb0f469f86660b2fa0bdadfd3ca585084"
Oct 29 22:45:22.902897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.902868    4559 generic.go:356] "Generic (PLEG): container finished" podID="2a0aee5f-3f63-4968-bb93-aec695b43215" containerID="20d6e21a00b1e7735b8f85a5dbcf9f69885339040bccb18d6a4dab75fc851f3a" exitCode=60
Oct 29 22:45:22.903380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.902950    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs" event={"ID":"2a0aee5f-3f63-4968-bb93-aec695b43215","Type":"ContainerDied","Data":"20d6e21a00b1e7735b8f85a5dbcf9f69885339040bccb18d6a4dab75fc851f3a"}
Oct 29 22:45:22.903552 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.903516    4559 scope.go:122] "RemoveContainer" containerID="20d6e21a00b1e7735b8f85a5dbcf9f69885339040bccb18d6a4dab75fc851f3a"
Oct 29 22:45:22.904025 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:22.904001    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs_odh-ai-gateway-infra(2a0aee5f-3f63-4968-bb93-aec695b43215)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs" podUID="2a0aee5f-3f63-4968-bb93-aec695b43215"
Oct 29 22:45:22.907141 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.907084    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p"
Oct 29 22:45:22.907271 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.907102    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p" event={"ID":"db5015fd-ba70-49ca-88aa-7f4f657ed546","Type":"ContainerDied","Data":"17f1e1461f8da8ca1977a8f6281c966f783d79a58d8cd6016ae6fe084dcb55f0"}
Oct 29 22:45:22.912638 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.912604    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr" event={"ID":"873fd28e-ffe4-4c7b-80d3-132892b4523e","Type":"ContainerDied","Data":"34be0641d777f7fec0e842e2ed89b87090655006e50b3c678faec9a0fca5d9ce"}
Oct 29 22:45:22.912774 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.912642    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr"
Oct 29 22:45:22.915066 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.915042    4559 scope.go:122] "RemoveContainer" containerID="8738c8f8bdbd274e38dc272eedb836e53227f039442b987a78e46565b62caa58"
Oct 29 22:45:22.920380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.920342    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx"
Oct 29 22:45:22.920924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.920338    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx" event={"ID":"698ca9e5-80e2-4475-a865-389166ac7ec9","Type":"ContainerDied","Data":"48279e187e58c8342284da88d5bad21b90a5a8adcc869296ea590bd2b77f47ce"}
Oct 29 22:45:22.933998 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.933962    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw" event={"ID":"cfaf0d72-b8d6-400c-a285-374310fdace7","Type":"ContainerDied","Data":"525425b86c24e711ca8dfdcdcca5e139adcedc7f2b5c94d6a9421e314a8a6ffb"}
Oct 29 22:45:22.934166 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.934113    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw"
Oct 29 22:45:22.942359 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.942335    4559 scope.go:122] "RemoveContainer" containerID="f5036d8ff2794fa195c9568099b1463bba523fff3f9cc089a78bd961bf5020af"
Oct 29 22:45:22.944799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.944767    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz" event={"ID":"6324171d-890b-481b-bf15-29dec862d313","Type":"ContainerStarted","Data":"95678cdadfbc590887284f665318bd5fef51d9cad8db34fff09e6006d0610fa3"}
Oct 29 22:45:22.946704 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.946681    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2"
Oct 29 22:45:22.946803 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.946761    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2" event={"ID":"254d613c-ac01-4d64-90d2-4c4630a1d9f1","Type":"ContainerDied","Data":"2db6bf1794f653d5c07579338943ac5d8a22df90765cbb83b122177c8a8bbbcc"}
Oct 29 22:45:22.946931 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.946861    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26" event={"ID":"bcd744cd-3522-40f3-9ef9-857fb1ab36e9","Type":"ContainerDied","Data":"271348d47686fa1937361e8f463c488a43f8c2c68a05150e6e1057a9a138f470"}
Oct 29 22:45:22.946931 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.946897    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k" event={"ID":"b1912dca-8e71-4a6c-ac38-934bd5f4a8df","Type":"ContainerDied","Data":"cd721eb8d8e31e01fa939c65d6eca77b8e9f5f5b682c8741d5c7081383730a79"}
Oct 29 22:45:22.949685 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.949655    4559 generic.go:356] "Generic (PLEG): container finished" podID="bb74ed25-e90c-48d8-bfb8-7687f250e690" containerID="7c2e5115470e28fe8add9290c2e9ff8875292bbb7d01475068fcfd4dabe99fbf" exitCode=60
Oct 29 22:45:22.949792 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.949731    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w" event={"ID":"bb74ed25-e90c-48d8-bfb8-7687f250e690","Type":"ContainerDied","Data":"7c2e5115470e28fe8add9290c2e9ff8875292bbb7d01475068fcfd4dabe99fbf"}
Oct 29 22:45:22.950124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.949875    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"
Oct 29 22:45:22.950324 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.950308    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"
Oct 29 22:45:22.950504 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.950482    4559 scope.go:122] "RemoveContainer" containerID="7c2e5115470e28fe8add9290c2e9ff8875292bbb7d01475068fcfd4dabe99fbf"
Oct 29 22:45:22.951034 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:22.951004    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w_odh-ai-gateway-infra(bb74ed25-e90c-48d8-bfb8-7687f250e690)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w" podUID="bb74ed25-e90c-48d8-bfb8-7687f250e690"
Oct 29 22:45:22.957955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.957901    4559 scope.go:122] "RemoveContainer" containerID="9022f1cff8a23951926b6ec40030ab9d1074d14f60933d8a6e05c5545c62ff38"
Oct 29 22:45:22.974993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.974966    4559 scope.go:122] "RemoveContainer" containerID="69558278c377c6931792cfbc59ed7c5d0c2455698887f6cafaed9eaef73621a8"
Oct 29 22:45:22.987144 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.987112    4559 scope.go:122] "RemoveContainer" containerID="32529d7af8ba0f78c08f8c04cd000b40c51ffa14e5d793eb1fa56fd761196d14"
Oct 29 22:45:22.998682 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.998634    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89"]
Oct 29 22:45:22.999159 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:22.999137    4559 scope.go:122] "RemoveContainer" containerID="9432453529b7313ff8b03010c425177a6b220c5df712b0025cf8affc20ae1588"
Oct 29 22:45:23.005972 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.005943    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e619b6-29888565-wmh89"]
Oct 29 22:45:23.016524 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.016501    4559 scope.go:122] "RemoveContainer" containerID="a31076e1163e36ce50059c341dde74a7694aa12d340d19bbe7d99fad7b1da4f0"
Oct 29 22:45:23.084529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.084494    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p"]
Oct 29 22:45:23.089928 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.089892    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888565-gfh9p"]
Oct 29 22:45:23.128197 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.128162    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2"]
Oct 29 22:45:23.142445 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.142409    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8d985a-29888565-qgqh2"]
Oct 29 22:45:23.207635 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.207543    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"]
Oct 29 22:45:23.212315 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.212285    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d3168-d94579868-m6l26"]
Oct 29 22:45:23.232505 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.232471    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="254d613c-ac01-4d64-90d2-4c4630a1d9f1" path="/var/lib/kubelet/pods/254d613c-ac01-4d64-90d2-4c4630a1d9f1/volumes"
Oct 29 22:45:23.233143 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.233125    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="450a7f81-0bb4-4c1d-9450-13845587b632" path="/var/lib/kubelet/pods/450a7f81-0bb4-4c1d-9450-13845587b632/volumes"
Oct 29 22:45:23.233794 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.233778    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="864f2e20-73e9-4552-a8bf-b44acebb7629" path="/var/lib/kubelet/pods/864f2e20-73e9-4552-a8bf-b44acebb7629/volumes"
Oct 29 22:45:23.234900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.234881    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd744cd-3522-40f3-9ef9-857fb1ab36e9" path="/var/lib/kubelet/pods/bcd744cd-3522-40f3-9ef9-857fb1ab36e9/volumes"
Oct 29 22:45:23.235471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.235452    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db5015fd-ba70-49ca-88aa-7f4f657ed546" path="/var/lib/kubelet/pods/db5015fd-ba70-49ca-88aa-7f4f657ed546/volumes"
Oct 29 22:45:23.255596 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.255562    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw"]
Oct 29 22:45:23.265039 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.265001    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888565-rvvfw"]
Oct 29 22:45:23.424303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.422887    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"]
Oct 29 22:45:23.429155 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.429109    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d3168-5cf56f8fd7-dbn2k"]
Oct 29 22:45:23.434021 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.433994    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr"]
Oct 29 22:45:23.438585 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.438557    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-05417d-29888565-jzghr"]
Oct 29 22:45:23.472038 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.471913    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx"]
Oct 29 22:45:23.478612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.478571    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"]
Oct 29 22:45:23.478916 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.478884    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2" podUID="8fad0702-72b6-4352-a752-f9aa98638048" containerName="istio-proxy" containerID="cri-o://111e1c2cce2fff8039e0e6d70f4d7adc552cf52942dba14a05720e06c5516a83" gracePeriod=30
Oct 29 22:45:23.484862 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.484796    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-06a95f-29888565-vqqgx"]
Oct 29 22:45:23.964579 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:23.964535    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz" event={"ID":"6324171d-890b-481b-bf15-29dec862d313","Type":"ContainerStarted","Data":"d6333c85feaa7c19946d1a62a9d0e9a1bf35c723db9976e4383fd97f6005462b"}
Oct 29 22:45:24.007942 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.007879    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz" podStartSLOduration=5.007860557 podStartE2EDuration="5.007860557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:24.004338121 +0000 UTC m=+1862.915781676" watchObservedRunningTime="2026-10-29 22:45:24.007860557 +0000 UTC m=+1862.919304117"
Oct 29 22:45:24.061508 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.061464    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs"]
Oct 29 22:45:24.256898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.256864    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs"
Oct 29 22:45:24.310919 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.310885    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0aee5f-3f63-4968-bb93-aec695b43215-kube-api-access-ps2kg\" (UniqueName: \"kubernetes.io/projected/2a0aee5f-3f63-4968-bb93-aec695b43215-kube-api-access-ps2kg\") pod \"2a0aee5f-3f63-4968-bb93-aec695b43215\" (UID: \"2a0aee5f-3f63-4968-bb93-aec695b43215\") "
Oct 29 22:45:24.311308 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.311282    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a0aee5f-3f63-4968-bb93-aec695b43215-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0aee5f-3f63-4968-bb93-aec695b43215-maas-api-service-ca\") pod \"2a0aee5f-3f63-4968-bb93-aec695b43215\" (UID: \"2a0aee5f-3f63-4968-bb93-aec695b43215\") "
Oct 29 22:45:24.311832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.311689    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0aee5f-3f63-4968-bb93-aec695b43215-maas-api-service-ca" pod "2a0aee5f-3f63-4968-bb93-aec695b43215" (UID: "2a0aee5f-3f63-4968-bb93-aec695b43215"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:24.312435 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.312292    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0aee5f-3f63-4968-bb93-aec695b43215-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:24.314907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.314204    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0aee5f-3f63-4968-bb93-aec695b43215-kube-api-access-ps2kg" pod "2a0aee5f-3f63-4968-bb93-aec695b43215" (UID: "2a0aee5f-3f63-4968-bb93-aec695b43215"). InnerVolumeSpecName "kube-api-access-ps2kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:24.319927 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.319877    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w"]
Oct 29 22:45:24.361678 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.361641    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"]
Oct 29 22:45:24.362859 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.362669    4559 cpu_manager.go:414] "containerMap: removing container" podUID="450a7f81-0bb4-4c1d-9450-13845587b632" containerName="cleanup"
Oct 29 22:45:24.363049 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.363016    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450a7f81-0bb4-4c1d-9450-13845587b632" containerName="cleanup"
Oct 29 22:45:24.363159 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.363149    4559 cpu_manager.go:414] "containerMap: removing container" podUID="873fd28e-ffe4-4c7b-80d3-132892b4523e" containerName="cleanup"
Oct 29 22:45:24.363215 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.363205    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="873fd28e-ffe4-4c7b-80d3-132892b4523e" containerName="cleanup"
Oct 29 22:45:24.363266 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.363259    4559 cpu_manager.go:414] "containerMap: removing container" podUID="254d613c-ac01-4d64-90d2-4c4630a1d9f1" containerName="cleanup"
Oct 29 22:45:24.363309 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.363303    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254d613c-ac01-4d64-90d2-4c4630a1d9f1" containerName="cleanup"
Oct 29 22:45:24.363372 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.363366    4559 cpu_manager.go:414] "containerMap: removing container" podUID="db5015fd-ba70-49ca-88aa-7f4f657ed546" containerName="cleanup"
Oct 29 22:45:24.363415 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.363409    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5015fd-ba70-49ca-88aa-7f4f657ed546" containerName="cleanup"
Oct 29 22:45:24.363464 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.363458    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b1912dca-8e71-4a6c-ac38-934bd5f4a8df" containerName="payload-pre-processing"
Oct 29 22:45:24.363512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.363505    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1912dca-8e71-4a6c-ac38-934bd5f4a8df" containerName="payload-pre-processing"
Oct 29 22:45:24.363572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.363565    4559 cpu_manager.go:414] "containerMap: removing container" podUID="254d613c-ac01-4d64-90d2-4c4630a1d9f1" containerName="cleanup"
Oct 29 22:45:24.363621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.363614    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254d613c-ac01-4d64-90d2-4c4630a1d9f1" containerName="cleanup"
Oct 29 22:45:24.363667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.363660    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2a0aee5f-3f63-4968-bb93-aec695b43215" containerName="cleanup"
Oct 29 22:45:24.363712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.363706    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0aee5f-3f63-4968-bb93-aec695b43215" containerName="cleanup"
Oct 29 22:45:24.363761 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.363755    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bcd744cd-3522-40f3-9ef9-857fb1ab36e9" containerName="payload-processing"
Oct 29 22:45:24.363835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.363803    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd744cd-3522-40f3-9ef9-857fb1ab36e9" containerName="payload-processing"
Oct 29 22:45:24.363931 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.363920    4559 cpu_manager.go:414] "containerMap: removing container" podUID="db5015fd-ba70-49ca-88aa-7f4f657ed546" containerName="cleanup"
Oct 29 22:45:24.364002 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.363992    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5015fd-ba70-49ca-88aa-7f4f657ed546" containerName="cleanup"
Oct 29 22:45:24.364072 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.364063    4559 cpu_manager.go:414] "containerMap: removing container" podUID="873fd28e-ffe4-4c7b-80d3-132892b4523e" containerName="cleanup"
Oct 29 22:45:24.364133 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.364125    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="873fd28e-ffe4-4c7b-80d3-132892b4523e" containerName="cleanup"
Oct 29 22:45:24.364202 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.364193    4559 cpu_manager.go:414] "containerMap: removing container" podUID="cfaf0d72-b8d6-400c-a285-374310fdace7" containerName="cleanup"
Oct 29 22:45:24.364269 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.364259    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfaf0d72-b8d6-400c-a285-374310fdace7" containerName="cleanup"
Oct 29 22:45:24.364331 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.364323    4559 cpu_manager.go:414] "containerMap: removing container" podUID="698ca9e5-80e2-4475-a865-389166ac7ec9" containerName="cleanup"
Oct 29 22:45:24.364401 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.364392    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698ca9e5-80e2-4475-a865-389166ac7ec9" containerName="cleanup"
Oct 29 22:45:24.364512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.364502    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2a0aee5f-3f63-4968-bb93-aec695b43215" containerName="cleanup"
Oct 29 22:45:24.364572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.364564    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0aee5f-3f63-4968-bb93-aec695b43215" containerName="cleanup"
Oct 29 22:45:24.364669 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.364639    4559 cpu_manager.go:414] "containerMap: removing container" podUID="cfaf0d72-b8d6-400c-a285-374310fdace7" containerName="cleanup"
Oct 29 22:45:24.364764 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.364751    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfaf0d72-b8d6-400c-a285-374310fdace7" containerName="cleanup"
Oct 29 22:45:24.364895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.364884    4559 cpu_manager.go:414] "containerMap: removing container" podUID="450a7f81-0bb4-4c1d-9450-13845587b632" containerName="cleanup"
Oct 29 22:45:24.364982 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.364973    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450a7f81-0bb4-4c1d-9450-13845587b632" containerName="cleanup"
Oct 29 22:45:24.365062 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365053    4559 cpu_manager.go:414] "containerMap: removing container" podUID="698ca9e5-80e2-4475-a865-389166ac7ec9" containerName="cleanup"
Oct 29 22:45:24.365130 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365121    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698ca9e5-80e2-4475-a865-389166ac7ec9" containerName="cleanup"
Oct 29 22:45:24.365203 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365194    4559 cpu_manager.go:414] "containerMap: removing container" podUID="db5015fd-ba70-49ca-88aa-7f4f657ed546" containerName="cleanup"
Oct 29 22:45:24.365278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365269    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5015fd-ba70-49ca-88aa-7f4f657ed546" containerName="cleanup"
Oct 29 22:45:24.365600 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365586    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="450a7f81-0bb4-4c1d-9450-13845587b632" containerName="cleanup"
Oct 29 22:45:24.365693 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365683    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="873fd28e-ffe4-4c7b-80d3-132892b4523e" containerName="cleanup"
Oct 29 22:45:24.365750 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365744    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="254d613c-ac01-4d64-90d2-4c4630a1d9f1" containerName="cleanup"
Oct 29 22:45:24.365827 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365802    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5015fd-ba70-49ca-88aa-7f4f657ed546" containerName="cleanup"
Oct 29 22:45:24.365894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365873    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1912dca-8e71-4a6c-ac38-934bd5f4a8df" containerName="payload-pre-processing"
Oct 29 22:45:24.365894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365891    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="254d613c-ac01-4d64-90d2-4c4630a1d9f1" containerName="cleanup"
Oct 29 22:45:24.365997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365902    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0aee5f-3f63-4968-bb93-aec695b43215" containerName="cleanup"
Oct 29 22:45:24.365997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365915    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd744cd-3522-40f3-9ef9-857fb1ab36e9" containerName="payload-processing"
Oct 29 22:45:24.365997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365929    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5015fd-ba70-49ca-88aa-7f4f657ed546" containerName="cleanup"
Oct 29 22:45:24.365997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365939    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="873fd28e-ffe4-4c7b-80d3-132892b4523e" containerName="cleanup"
Oct 29 22:45:24.365997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365950    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="450a7f81-0bb4-4c1d-9450-13845587b632" containerName="cleanup"
Oct 29 22:45:24.365997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365961    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfaf0d72-b8d6-400c-a285-374310fdace7" containerName="cleanup"
Oct 29 22:45:24.365997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365969    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="698ca9e5-80e2-4475-a865-389166ac7ec9" containerName="cleanup"
Oct 29 22:45:24.365997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.365989    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0aee5f-3f63-4968-bb93-aec695b43215" containerName="cleanup"
Oct 29 22:45:24.366357 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.366002    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfaf0d72-b8d6-400c-a285-374310fdace7" containerName="cleanup"
Oct 29 22:45:24.366357 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.366012    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="698ca9e5-80e2-4475-a865-389166ac7ec9" containerName="cleanup"
Oct 29 22:45:24.366357 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.366023    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="864f2e20-73e9-4552-a8bf-b44acebb7629" containerName="cleanup"
Oct 29 22:45:24.366357 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.366032    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="450a7f81-0bb4-4c1d-9450-13845587b632" containerName="cleanup"
Oct 29 22:45:24.366357 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.366041    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="698ca9e5-80e2-4475-a865-389166ac7ec9" containerName="cleanup"
Oct 29 22:45:24.366357 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.366052    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfaf0d72-b8d6-400c-a285-374310fdace7" containerName="cleanup"
Oct 29 22:45:24.375752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.374086    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.386962 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.386933    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"]
Oct 29 22:45:24.394588 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.394534    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5a69c9\"/\"test-model-5a69c9-kserve-self-signed-certs\""
Oct 29 22:45:24.395109 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.394914    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5a69c9\"/\"openshift-service-ca.crt\""
Oct 29 22:45:24.395278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.394955    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5a69c9\"/\"default-dockercfg-brddh\""
Oct 29 22:45:24.395493 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.395069    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5a69c9\"/\"kube-root-ca.crt\""
Oct 29 22:45:24.413917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.413877    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9779fead-40ad-4cc9-b05e-d2a8d670af27-tls-certs\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.414154 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.414134    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-tmp-dir\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.414461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.414398    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmh5q\" (UniqueName: \"kubernetes.io/projected/9779fead-40ad-4cc9-b05e-d2a8d670af27-kube-api-access-fmh5q\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.414615 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.414602    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-model-cache\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.414896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.414883    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-home\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.415088 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.415068    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-dshm\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.415358 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.415346    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps2kg\" (UniqueName: \"kubernetes.io/projected/2a0aee5f-3f63-4968-bb93-aec695b43215-kube-api-access-ps2kg\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:24.517392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.517320    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-tmp-dir\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.517549 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.517461    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmh5q\" (UniqueName: \"kubernetes.io/projected/9779fead-40ad-4cc9-b05e-d2a8d670af27-kube-api-access-fmh5q\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.517549 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.517499    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-model-cache\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.517641 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.517583    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-home\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.517641 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.517629    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-dshm\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.517714 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.517704    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9779fead-40ad-4cc9-b05e-d2a8d670af27-tls-certs\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.518793 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.518766    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-tmp-dir\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.519081 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.518857    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-home\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.520950 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.520916    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-model-cache\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.521074 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.521055    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9779fead-40ad-4cc9-b05e-d2a8d670af27-tls-certs\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.521119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.521080    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-dshm\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.569837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.569222    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w"
Oct 29 22:45:24.649319 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.629613    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmh5q\" (UniqueName: \"kubernetes.io/projected/9779fead-40ad-4cc9-b05e-d2a8d670af27-kube-api-access-fmh5q\") pod \"test-model-5a69c9-kserve-769b54b4b5-d9wlb\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") " pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.649319 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.631431    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb74ed25-e90c-48d8-bfb8-7687f250e690-kube-api-access-jzx78\" (UniqueName: \"kubernetes.io/projected/bb74ed25-e90c-48d8-bfb8-7687f250e690-kube-api-access-jzx78\") pod \"bb74ed25-e90c-48d8-bfb8-7687f250e690\" (UID: \"bb74ed25-e90c-48d8-bfb8-7687f250e690\") "
Oct 29 22:45:24.649319 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.631490    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb74ed25-e90c-48d8-bfb8-7687f250e690-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb74ed25-e90c-48d8-bfb8-7687f250e690-maas-api-service-ca\") pod \"bb74ed25-e90c-48d8-bfb8-7687f250e690\" (UID: \"bb74ed25-e90c-48d8-bfb8-7687f250e690\") "
Oct 29 22:45:24.688932 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.688539    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb74ed25-e90c-48d8-bfb8-7687f250e690-kube-api-access-jzx78" pod "bb74ed25-e90c-48d8-bfb8-7687f250e690" (UID: "bb74ed25-e90c-48d8-bfb8-7687f250e690"). InnerVolumeSpecName "kube-api-access-jzx78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:24.721852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.708655    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:24.739849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.739755    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzx78\" (UniqueName: \"kubernetes.io/projected/bb74ed25-e90c-48d8-bfb8-7687f250e690-kube-api-access-jzx78\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:24.783005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.782683    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb74ed25-e90c-48d8-bfb8-7687f250e690-maas-api-service-ca" pod "bb74ed25-e90c-48d8-bfb8-7687f250e690" (UID: "bb74ed25-e90c-48d8-bfb8-7687f250e690"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:24.841281 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.841246    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb74ed25-e90c-48d8-bfb8-7687f250e690-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:24.990605 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.990571    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs"
Oct 29 22:45:24.991161 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.990574    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs" event={"ID":"2a0aee5f-3f63-4968-bb93-aec695b43215","Type":"ContainerDied","Data":"ddc0d8cdb703c8e9dfb2f65495419bc9123cda1b556f8c62ce6c568841f4f732"}
Oct 29 22:45:24.991161 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.990718    4559 scope.go:122] "RemoveContainer" containerID="20d6e21a00b1e7735b8f85a5dbcf9f69885339040bccb18d6a4dab75fc851f3a"
Oct 29 22:45:24.998022 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.997991    4559 generic.go:356] "Generic (PLEG): container finished" podID="6324171d-890b-481b-bf15-29dec862d313" containerID="d6333c85feaa7c19946d1a62a9d0e9a1bf35c723db9976e4383fd97f6005462b" exitCode=0
Oct 29 22:45:24.998295 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:24.998227    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz" event={"ID":"6324171d-890b-481b-bf15-29dec862d313","Type":"ContainerDied","Data":"d6333c85feaa7c19946d1a62a9d0e9a1bf35c723db9976e4383fd97f6005462b"}
Oct 29 22:45:25.000276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:25.000243    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w" event={"ID":"bb74ed25-e90c-48d8-bfb8-7687f250e690","Type":"ContainerDied","Data":"4b6313c9feb885ff936c22a02c0363150d26496ed6c1f4d2bfa18b45035723a3"}
Oct 29 22:45:25.000392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:25.000320    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w"
Oct 29 22:45:25.018598 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:25.018570    4559 scope.go:122] "RemoveContainer" containerID="7c2e5115470e28fe8add9290c2e9ff8875292bbb7d01475068fcfd4dabe99fbf"
Oct 29 22:45:25.103618 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:25.103532    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w"]
Oct 29 22:45:25.109107 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:25.109071    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888565-74r5w"]
Oct 29 22:45:25.141001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:25.140962    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs"]
Oct 29 22:45:25.146624 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:25.146589    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888565-7gqjs"]
Oct 29 22:45:25.155214 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:25.155181    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"]
Oct 29 22:45:25.155375 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:45:25.155323    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9779fead_40ad_4cc9_b05e_d2a8d670af27.slice/crio-5237e380f116b72354e9020d663e0eb3295224cb76e645fc3c120c437d87ddba WatchSource:0}: Error finding container 5237e380f116b72354e9020d663e0eb3295224cb76e645fc3c120c437d87ddba: Status 404 returned error can't find the container with id 5237e380f116b72354e9020d663e0eb3295224cb76e645fc3c120c437d87ddba
Oct 29 22:45:25.257507 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:25.257466    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a0aee5f-3f63-4968-bb93-aec695b43215" path="/var/lib/kubelet/pods/2a0aee5f-3f63-4968-bb93-aec695b43215/volumes"
Oct 29 22:45:25.258530 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:25.258502    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="698ca9e5-80e2-4475-a865-389166ac7ec9" path="/var/lib/kubelet/pods/698ca9e5-80e2-4475-a865-389166ac7ec9/volumes"
Oct 29 22:45:25.259173 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:25.259155    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="873fd28e-ffe4-4c7b-80d3-132892b4523e" path="/var/lib/kubelet/pods/873fd28e-ffe4-4c7b-80d3-132892b4523e/volumes"
Oct 29 22:45:25.260234 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:25.260212    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1912dca-8e71-4a6c-ac38-934bd5f4a8df" path="/var/lib/kubelet/pods/b1912dca-8e71-4a6c-ac38-934bd5f4a8df/volumes"
Oct 29 22:45:25.260789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:25.260769    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb74ed25-e90c-48d8-bfb8-7687f250e690" path="/var/lib/kubelet/pods/bb74ed25-e90c-48d8-bfb8-7687f250e690/volumes"
Oct 29 22:45:25.261559 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:25.261539    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfaf0d72-b8d6-400c-a285-374310fdace7" path="/var/lib/kubelet/pods/cfaf0d72-b8d6-400c-a285-374310fdace7/volumes"
Oct 29 22:45:26.015086 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:26.015038    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb" event={"ID":"9779fead-40ad-4cc9-b05e-d2a8d670af27","Type":"ContainerStarted","Data":"f239e2e6b1b4c0de5523e931793579956097b135f8198ce44ffdbc26c2b37840"}
Oct 29 22:45:26.015577 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:26.015097    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb" event={"ID":"9779fead-40ad-4cc9-b05e-d2a8d670af27","Type":"ContainerStarted","Data":"5237e380f116b72354e9020d663e0eb3295224cb76e645fc3c120c437d87ddba"}
Oct 29 22:45:26.059715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:26.059599    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb" podStartSLOduration=2.059582825 podStartE2EDuration="2.059582825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:26.055270956 +0000 UTC m=+1864.966714513" watchObservedRunningTime="2026-10-29 22:45:26.059582825 +0000 UTC m=+1864.971026438"
Oct 29 22:45:26.347642 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:26.347554    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"
Oct 29 22:45:26.406305 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:26.406266    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6324171d-890b-481b-bf15-29dec862d313-kube-api-access-sfw48\" (UniqueName: \"kubernetes.io/projected/6324171d-890b-481b-bf15-29dec862d313-kube-api-access-sfw48\") pod \"6324171d-890b-481b-bf15-29dec862d313\" (UID: \"6324171d-890b-481b-bf15-29dec862d313\") "
Oct 29 22:45:26.406477 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:26.406347    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6324171d-890b-481b-bf15-29dec862d313-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6324171d-890b-481b-bf15-29dec862d313-maas-api-xks-ca\") pod \"6324171d-890b-481b-bf15-29dec862d313\" (UID: \"6324171d-890b-481b-bf15-29dec862d313\") "
Oct 29 22:45:26.406477 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:26.406443    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6324171d-890b-481b-bf15-29dec862d313-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6324171d-890b-481b-bf15-29dec862d313-maas-api-openshift-ca\") pod \"6324171d-890b-481b-bf15-29dec862d313\" (UID: \"6324171d-890b-481b-bf15-29dec862d313\") "
Oct 29 22:45:26.406885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:26.406839    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6324171d-890b-481b-bf15-29dec862d313-maas-api-openshift-ca" pod "6324171d-890b-481b-bf15-29dec862d313" (UID: "6324171d-890b-481b-bf15-29dec862d313"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:26.407411 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:26.407381    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6324171d-890b-481b-bf15-29dec862d313-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:26.410433 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:26.410402    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6324171d-890b-481b-bf15-29dec862d313-kube-api-access-sfw48" pod "6324171d-890b-481b-bf15-29dec862d313" (UID: "6324171d-890b-481b-bf15-29dec862d313"). InnerVolumeSpecName "kube-api-access-sfw48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:26.412287 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:26.412239    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6324171d-890b-481b-bf15-29dec862d313-maas-api-xks-ca" pod "6324171d-890b-481b-bf15-29dec862d313" (UID: "6324171d-890b-481b-bf15-29dec862d313"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:45:26.509097 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:26.509051    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfw48\" (UniqueName: \"kubernetes.io/projected/6324171d-890b-481b-bf15-29dec862d313-kube-api-access-sfw48\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:26.509097 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:26.509088    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6324171d-890b-481b-bf15-29dec862d313-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:26.724258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:26.724222    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:26.731366 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:26.731327    4559 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:45:26.731576 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:26.731392    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm" podUID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:45:27.034684 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.034641    4559 generic.go:356] "Generic (PLEG): container finished" podID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" containerID="9dd62e009a5921088740e5c1727fd5a7cb015c0b386742ecaf8e17ceb7c4646f" exitCode=0
Oct 29 22:45:27.035209 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.034846    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm" event={"ID":"158bafa6-a6ed-49d4-8ac2-2722c7be5efb","Type":"ContainerDied","Data":"9dd62e009a5921088740e5c1727fd5a7cb015c0b386742ecaf8e17ceb7c4646f"}
Oct 29 22:45:27.035209 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.034927    4559 scope.go:122] "RemoveContainer" containerID="f230b8fda8326c414997183af033d54402c09d6b13c91bb439562b079482c444"
Oct 29 22:45:27.145310 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.145256    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"
Oct 29 22:45:27.149395 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.148416    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz" event={"ID":"6324171d-890b-481b-bf15-29dec862d313","Type":"ContainerDied","Data":"95678cdadfbc590887284f665318bd5fef51d9cad8db34fff09e6006d0610fa3"}
Oct 29 22:45:27.150116 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.150075    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95678cdadfbc590887284f665318bd5fef51d9cad8db34fff09e6006d0610fa3"
Oct 29 22:45:27.184469 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.184446    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:45:27.256254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.255400    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-credential-socket\") pod \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") "
Oct 29 22:45:27.256254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.255661    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-kube-api-access-8ljkb\" (UniqueName: \"kubernetes.io/projected/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-kube-api-access-8ljkb\") pod \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") "
Oct 29 22:45:27.256254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.255760    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istiod-ca-cert\") pod \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") "
Oct 29 22:45:27.256254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.255828    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-podinfo\") pod \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") "
Oct 29 22:45:27.256254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.255921    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-workload-certs\") pod \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") "
Oct 29 22:45:27.256254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.255949    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-token\" (UniqueName: \"kubernetes.io/projected/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-token\") pod \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") "
Oct 29 22:45:27.256254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.255998    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-data\") pod \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") "
Oct 29 22:45:27.256254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.256077    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-envoy\") pod \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") "
Oct 29 22:45:27.256254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.256115    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-workload-socket\") pod \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\" (UID: \"158bafa6-a6ed-49d4-8ac2-2722c7be5efb\") "
Oct 29 22:45:27.258591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.257760    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-credential-socket" pod "158bafa6-a6ed-49d4-8ac2-2722c7be5efb" (UID: "158bafa6-a6ed-49d4-8ac2-2722c7be5efb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:27.259731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.259696    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-data" pod "158bafa6-a6ed-49d4-8ac2-2722c7be5efb" (UID: "158bafa6-a6ed-49d4-8ac2-2722c7be5efb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:27.259951 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.259932    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-workload-certs" pod "158bafa6-a6ed-49d4-8ac2-2722c7be5efb" (UID: "158bafa6-a6ed-49d4-8ac2-2722c7be5efb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:27.260204 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.260177    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-workload-socket" pod "158bafa6-a6ed-49d4-8ac2-2722c7be5efb" (UID: "158bafa6-a6ed-49d4-8ac2-2722c7be5efb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:27.260589 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.260562    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istiod-ca-cert" pod "158bafa6-a6ed-49d4-8ac2-2722c7be5efb" (UID: "158bafa6-a6ed-49d4-8ac2-2722c7be5efb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:27.263380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.261773    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-kube-api-access-8ljkb" pod "158bafa6-a6ed-49d4-8ac2-2722c7be5efb" (UID: "158bafa6-a6ed-49d4-8ac2-2722c7be5efb"). InnerVolumeSpecName "kube-api-access-8ljkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:27.263380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.262007    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-envoy" pod "158bafa6-a6ed-49d4-8ac2-2722c7be5efb" (UID: "158bafa6-a6ed-49d4-8ac2-2722c7be5efb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:27.269079 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.263800    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-token" pod "158bafa6-a6ed-49d4-8ac2-2722c7be5efb" (UID: "158bafa6-a6ed-49d4-8ac2-2722c7be5efb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:27.269079 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.263956    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-podinfo" pod "158bafa6-a6ed-49d4-8ac2-2722c7be5efb" (UID: "158bafa6-a6ed-49d4-8ac2-2722c7be5efb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:45:27.359085 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.359043    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ljkb\" (UniqueName: \"kubernetes.io/projected/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-kube-api-access-8ljkb\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:27.359085 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.359086    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:27.359303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.359101    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:27.359303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.359114    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:27.359303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.359125    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:27.359303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.359137    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:27.359303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.359150    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:27.359303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.359163    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:27.359303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:27.359177    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158bafa6-a6ed-49d4-8ac2-2722c7be5efb-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:28.157781 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.157745    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"
Oct 29 22:45:28.157781 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.157768    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm" event={"ID":"158bafa6-a6ed-49d4-8ac2-2722c7be5efb","Type":"ContainerDied","Data":"139e6b8a09e34018a7150367c795c7df714153ed4e4df76820b20e5522ea8390"}
Oct 29 22:45:28.158389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.157871    4559 scope.go:122] "RemoveContainer" containerID="9dd62e009a5921088740e5c1727fd5a7cb015c0b386742ecaf8e17ceb7c4646f"
Oct 29 22:45:28.246292 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.246248    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"]
Oct 29 22:45:28.259334 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.259290    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-l4blm"]
Oct 29 22:45:28.891867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.891051    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:45:28.995059 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.995009    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-istio-data\") pod \"8fad0702-72b6-4352-a752-f9aa98638048\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") "
Oct 29 22:45:28.995417 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.995390    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-workload-socket\") pod \"8fad0702-72b6-4352-a752-f9aa98638048\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") "
Oct 29 22:45:28.995782 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.995615    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fad0702-72b6-4352-a752-f9aa98638048-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fad0702-72b6-4352-a752-f9aa98638048-istio-podinfo\") pod \"8fad0702-72b6-4352-a752-f9aa98638048\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") "
Oct 29 22:45:28.995782 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.995706    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-workload-certs\") pod \"8fad0702-72b6-4352-a752-f9aa98638048\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") "
Oct 29 22:45:28.996416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.996030    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-workload-certs" pod "8fad0702-72b6-4352-a752-f9aa98638048" (UID: "8fad0702-72b6-4352-a752-f9aa98638048"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:28.996416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.996145    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-istio-envoy\") pod \"8fad0702-72b6-4352-a752-f9aa98638048\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") "
Oct 29 22:45:28.996416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.996199    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fad0702-72b6-4352-a752-f9aa98638048-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fad0702-72b6-4352-a752-f9aa98638048-istiod-ca-cert\") pod \"8fad0702-72b6-4352-a752-f9aa98638048\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") "
Oct 29 22:45:28.996416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.996285    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fad0702-72b6-4352-a752-f9aa98638048-kube-api-access-4kmxp\" (UniqueName: \"kubernetes.io/projected/8fad0702-72b6-4352-a752-f9aa98638048-kube-api-access-4kmxp\") pod \"8fad0702-72b6-4352-a752-f9aa98638048\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") "
Oct 29 22:45:28.996416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.996368    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fad0702-72b6-4352-a752-f9aa98638048-istio-token\" (UniqueName: \"kubernetes.io/projected/8fad0702-72b6-4352-a752-f9aa98638048-istio-token\") pod \"8fad0702-72b6-4352-a752-f9aa98638048\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") "
Oct 29 22:45:28.996714 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.996449    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-credential-socket\") pod \"8fad0702-72b6-4352-a752-f9aa98638048\" (UID: \"8fad0702-72b6-4352-a752-f9aa98638048\") "
Oct 29 22:45:28.996775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.996727    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-istio-data" pod "8fad0702-72b6-4352-a752-f9aa98638048" (UID: "8fad0702-72b6-4352-a752-f9aa98638048"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:28.997268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.997059    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-workload-socket" pod "8fad0702-72b6-4352-a752-f9aa98638048" (UID: "8fad0702-72b6-4352-a752-f9aa98638048"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:28.997382 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.997297    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-credential-socket" pod "8fad0702-72b6-4352-a752-f9aa98638048" (UID: "8fad0702-72b6-4352-a752-f9aa98638048"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:28.997712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.997625    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:28.997712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.997652    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:28.997712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.997668    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:28.997712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.997679    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:28.998361 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:28.998340    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fad0702-72b6-4352-a752-f9aa98638048-istiod-ca-cert" pod "8fad0702-72b6-4352-a752-f9aa98638048" (UID: "8fad0702-72b6-4352-a752-f9aa98638048"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:29.002368 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.002120    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-istio-envoy" pod "8fad0702-72b6-4352-a752-f9aa98638048" (UID: "8fad0702-72b6-4352-a752-f9aa98638048"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:29.002368 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.002325    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fad0702-72b6-4352-a752-f9aa98638048-istio-podinfo" pod "8fad0702-72b6-4352-a752-f9aa98638048" (UID: "8fad0702-72b6-4352-a752-f9aa98638048"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:45:29.014854 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.005864    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fad0702-72b6-4352-a752-f9aa98638048-kube-api-access-4kmxp" pod "8fad0702-72b6-4352-a752-f9aa98638048" (UID: "8fad0702-72b6-4352-a752-f9aa98638048"). InnerVolumeSpecName "kube-api-access-4kmxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:29.015404 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.015026    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fad0702-72b6-4352-a752-f9aa98638048-istio-token" pod "8fad0702-72b6-4352-a752-f9aa98638048" (UID: "8fad0702-72b6-4352-a752-f9aa98638048"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:29.019507 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.019476    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"]
Oct 29 22:45:29.020947 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.020922    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6324171d-890b-481b-bf15-29dec862d313" containerName="revoke-keys"
Oct 29 22:45:29.021089 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.021073    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6324171d-890b-481b-bf15-29dec862d313" containerName="revoke-keys"
Oct 29 22:45:29.021187 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.021175    4559 cpu_manager.go:414] "containerMap: removing container" podUID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" containerName="istio-proxy"
Oct 29 22:45:29.021264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.021253    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" containerName="istio-proxy"
Oct 29 22:45:29.021387 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.021375    4559 cpu_manager.go:414] "containerMap: removing container" podUID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" containerName="istio-proxy"
Oct 29 22:45:29.021476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.021465    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" containerName="istio-proxy"
Oct 29 22:45:29.021608 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.021595    4559 cpu_manager.go:414] "containerMap: removing container" podUID="450a7f81-0bb4-4c1d-9450-13845587b632" containerName="cleanup"
Oct 29 22:45:29.021690 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.021680    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450a7f81-0bb4-4c1d-9450-13845587b632" containerName="cleanup"
Oct 29 22:45:29.021790 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.021780    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bb74ed25-e90c-48d8-bfb8-7687f250e690" containerName="cleanup"
Oct 29 22:45:29.021917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.021903    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb74ed25-e90c-48d8-bfb8-7687f250e690" containerName="cleanup"
Oct 29 22:45:29.022013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.022002    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bb74ed25-e90c-48d8-bfb8-7687f250e690" containerName="cleanup"
Oct 29 22:45:29.022095 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.022085    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb74ed25-e90c-48d8-bfb8-7687f250e690" containerName="cleanup"
Oct 29 22:45:29.022182 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.022171    4559 cpu_manager.go:414] "containerMap: removing container" podUID="254d613c-ac01-4d64-90d2-4c4630a1d9f1" containerName="cleanup"
Oct 29 22:45:29.022258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.022248    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254d613c-ac01-4d64-90d2-4c4630a1d9f1" containerName="cleanup"
Oct 29 22:45:29.022340 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.022331    4559 cpu_manager.go:414] "containerMap: removing container" podUID="873fd28e-ffe4-4c7b-80d3-132892b4523e" containerName="cleanup"
Oct 29 22:45:29.022413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.022403    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="873fd28e-ffe4-4c7b-80d3-132892b4523e" containerName="cleanup"
Oct 29 22:45:29.022504 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.022493    4559 cpu_manager.go:414] "containerMap: removing container" podUID="cfaf0d72-b8d6-400c-a285-374310fdace7" containerName="cleanup"
Oct 29 22:45:29.022588 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.022577    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfaf0d72-b8d6-400c-a285-374310fdace7" containerName="cleanup"
Oct 29 22:45:29.022670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.022661    4559 cpu_manager.go:414] "containerMap: removing container" podUID="698ca9e5-80e2-4475-a865-389166ac7ec9" containerName="cleanup"
Oct 29 22:45:29.022742 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.022732    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698ca9e5-80e2-4475-a865-389166ac7ec9" containerName="cleanup"
Oct 29 22:45:29.022871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.022859    4559 cpu_manager.go:414] "containerMap: removing container" podUID="8fad0702-72b6-4352-a752-f9aa98638048" containerName="istio-proxy"
Oct 29 22:45:29.022957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.022945    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fad0702-72b6-4352-a752-f9aa98638048" containerName="istio-proxy"
Oct 29 22:45:29.023058 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.023048    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2a0aee5f-3f63-4968-bb93-aec695b43215" containerName="cleanup"
Oct 29 22:45:29.023140 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.023130    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0aee5f-3f63-4968-bb93-aec695b43215" containerName="cleanup"
Oct 29 22:45:29.023463 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.023444    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fad0702-72b6-4352-a752-f9aa98638048" containerName="istio-proxy"
Oct 29 22:45:29.023605 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.023590    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb74ed25-e90c-48d8-bfb8-7687f250e690" containerName="cleanup"
Oct 29 22:45:29.023695 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.023685    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6324171d-890b-481b-bf15-29dec862d313" containerName="revoke-keys"
Oct 29 22:45:29.023783 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.023774    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" containerName="istio-proxy"
Oct 29 22:45:29.023898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.023885    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" containerName="istio-proxy"
Oct 29 22:45:29.024006 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.023996    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb74ed25-e90c-48d8-bfb8-7687f250e690" containerName="cleanup"
Oct 29 22:45:29.024085 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.024076    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb74ed25-e90c-48d8-bfb8-7687f250e690" containerName="cleanup"
Oct 29 22:45:29.024167 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.024158    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="254d613c-ac01-4d64-90d2-4c4630a1d9f1" containerName="cleanup"
Oct 29 22:45:29.024249 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.024240    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0aee5f-3f63-4968-bb93-aec695b43215" containerName="cleanup"
Oct 29 22:45:29.024321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.024311    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5015fd-ba70-49ca-88aa-7f4f657ed546" containerName="cleanup"
Oct 29 22:45:29.024396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.024386    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="873fd28e-ffe4-4c7b-80d3-132892b4523e" containerName="cleanup"
Oct 29 22:45:29.029363 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.029341    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.039611 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.039580    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"]
Oct 29 22:45:29.045883 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.045854    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-05417d\"/\"kube-root-ca.crt\""
Oct 29 22:45:29.046169 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.046151    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-05417d\"/\"openshift-service-ca.crt\""
Oct 29 22:45:29.046402 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.046385    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-05417d\"/\"default-dockercfg-4cshh\""
Oct 29 22:45:29.046624 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.046607    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-05417d\"/\"test-model-05417d-kserve-self-signed-certs\""
Oct 29 22:45:29.103180 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.103146    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fad0702-72b6-4352-a752-f9aa98638048-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:29.103180 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.103181    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fad0702-72b6-4352-a752-f9aa98638048-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:29.103359 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.103195    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fad0702-72b6-4352-a752-f9aa98638048-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:29.103359 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.103210    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kmxp\" (UniqueName: \"kubernetes.io/projected/8fad0702-72b6-4352-a752-f9aa98638048-kube-api-access-4kmxp\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:29.103359 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.103224    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fad0702-72b6-4352-a752-f9aa98638048-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:29.172673 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.172629    4559 generic.go:356] "Generic (PLEG): container finished" podID="8fad0702-72b6-4352-a752-f9aa98638048" containerID="111e1c2cce2fff8039e0e6d70f4d7adc552cf52942dba14a05720e06c5516a83" exitCode=0
Oct 29 22:45:29.173184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.172683    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"
Oct 29 22:45:29.173184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.172709    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2" event={"ID":"8fad0702-72b6-4352-a752-f9aa98638048","Type":"ContainerDied","Data":"111e1c2cce2fff8039e0e6d70f4d7adc552cf52942dba14a05720e06c5516a83"}
Oct 29 22:45:29.173184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.172802    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2" event={"ID":"8fad0702-72b6-4352-a752-f9aa98638048","Type":"ContainerDied","Data":"ce4dd3157a2428547fa7296452e77c91ceee3784c03217090fcc79084c642767"}
Oct 29 22:45:29.173184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.172856    4559 scope.go:122] "RemoveContainer" containerID="111e1c2cce2fff8039e0e6d70f4d7adc552cf52942dba14a05720e06c5516a83"
Oct 29 22:45:29.185413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.185390    4559 scope.go:122] "RemoveContainer" containerID="111e1c2cce2fff8039e0e6d70f4d7adc552cf52942dba14a05720e06c5516a83"
Oct 29 22:45:29.185770 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:29.185750    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111e1c2cce2fff8039e0e6d70f4d7adc552cf52942dba14a05720e06c5516a83\": container with ID starting with 111e1c2cce2fff8039e0e6d70f4d7adc552cf52942dba14a05720e06c5516a83 not found: ID does not exist" containerID="111e1c2cce2fff8039e0e6d70f4d7adc552cf52942dba14a05720e06c5516a83"
Oct 29 22:45:29.185880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.185786    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111e1c2cce2fff8039e0e6d70f4d7adc552cf52942dba14a05720e06c5516a83"} err="failed to get container status \"111e1c2cce2fff8039e0e6d70f4d7adc552cf52942dba14a05720e06c5516a83\": rpc error: code = NotFound desc = could not find container \"111e1c2cce2fff8039e0e6d70f4d7adc552cf52942dba14a05720e06c5516a83\": container with ID starting with 111e1c2cce2fff8039e0e6d70f4d7adc552cf52942dba14a05720e06c5516a83 not found: ID does not exist"
Oct 29 22:45:29.204276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.204236    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-home\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.204461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.204288    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-tmp-dir\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.204461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.204359    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03d29ddc-3695-4fda-ae9d-fb32b252fce0-tls-certs\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.204580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.204480    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-dshm\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.204580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.204500    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-model-cache\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.204658 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.204592    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t95nk\" (UniqueName: \"kubernetes.io/projected/03d29ddc-3695-4fda-ae9d-fb32b252fce0-kube-api-access-t95nk\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.232108 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.232070    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158bafa6-a6ed-49d4-8ac2-2722c7be5efb" path="/var/lib/kubelet/pods/158bafa6-a6ed-49d4-8ac2-2722c7be5efb/volumes"
Oct 29 22:45:29.233893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.232573    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"]
Oct 29 22:45:29.247845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.244535    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8npq2"]
Oct 29 22:45:29.307890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.307148    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-home\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.307890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.307238    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-tmp-dir\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.307890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.307276    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03d29ddc-3695-4fda-ae9d-fb32b252fce0-tls-certs\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.307890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.307391    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-dshm\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.307890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.307416    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-model-cache\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.307890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.307517    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t95nk\" (UniqueName: \"kubernetes.io/projected/03d29ddc-3695-4fda-ae9d-fb32b252fce0-kube-api-access-t95nk\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.311837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.311708    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-home\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.312026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.311990    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-tmp-dir\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.327893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.324129    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-model-cache\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.338835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.338159    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-dshm\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.354837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.346300    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03d29ddc-3695-4fda-ae9d-fb32b252fce0-tls-certs\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.366341 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.363973    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t95nk\" (UniqueName: \"kubernetes.io/projected/03d29ddc-3695-4fda-ae9d-fb32b252fce0-kube-api-access-t95nk\") pod \"test-model-05417d-kserve-86d6967fcc-rtjml\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") " pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.660525 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.660480    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:29.904541 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.904507    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"]
Oct 29 22:45:29.998055 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:29.990474    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"]
Oct 29 22:45:29.998055 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:45:29.993046    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03d29ddc_3695_4fda_ae9d_fb32b252fce0.slice/crio-c64b058eca8bdd4282e3c16202833d206acd40dce8b228f24d35f6979546ce42 WatchSource:0}: Error finding container c64b058eca8bdd4282e3c16202833d206acd40dce8b228f24d35f6979546ce42: Status 404 returned error can't find the container with id c64b058eca8bdd4282e3c16202833d206acd40dce8b228f24d35f6979546ce42
Oct 29 22:45:30.261618 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:30.260791    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml" event={"ID":"03d29ddc-3695-4fda-ae9d-fb32b252fce0","Type":"ContainerStarted","Data":"46e3cffe587c47a1d27d3e716beec6cd49d340a514438c6d975d6a372d63d177"}
Oct 29 22:45:30.261618 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:30.260859    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml" event={"ID":"03d29ddc-3695-4fda-ae9d-fb32b252fce0","Type":"ContainerStarted","Data":"c64b058eca8bdd4282e3c16202833d206acd40dce8b228f24d35f6979546ce42"}
Oct 29 22:45:30.309747 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:30.309681    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml" podStartSLOduration=2.309661375 podStartE2EDuration="2.309661375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:30.304366379 +0000 UTC m=+1869.215809934" watchObservedRunningTime="2026-10-29 22:45:30.309661375 +0000 UTC m=+1869.221104933"
Oct 29 22:45:30.680839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:30.678501    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:31.232000 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:31.231958    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fad0702-72b6-4352-a752-f9aa98638048" path="/var/lib/kubelet/pods/8fad0702-72b6-4352-a752-f9aa98638048/volumes"
Oct 29 22:45:34.724563 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:34.724528    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:34.739920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:34.739885    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:34.905134 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:34.905068    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg" podUID="83c4474e-4311-4cc6-a79f-aa4bc4edfcb0" containerName="payload-pre-processing" containerID="cri-o://955dac93fb7a24f9d879fea41427f2aef7e1fbd8ee1b17ad7a8fa70f2af9eb89" gracePeriod=25
Oct 29 22:45:35.316027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:35.315992    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:45:39.557882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.557843    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"]
Oct 29 22:45:39.558624 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.558606    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bb74ed25-e90c-48d8-bfb8-7687f250e690" containerName="cleanup"
Oct 29 22:45:39.558624 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.558621    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb74ed25-e90c-48d8-bfb8-7687f250e690" containerName="cleanup"
Oct 29 22:45:39.564278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.564252    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.593125 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.593095    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e619b6-openshift-default-dockercfg-rrqdd\""
Oct 29 22:45:39.596883 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.596800    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"]
Oct 29 22:45:39.661461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.661425    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:39.679452 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.679423    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:39.685675 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.685638    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istiod-ca-cert\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.685869 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.685746    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-podinfo\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.685976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.685949    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-workload-certs\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.686047 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.685995    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-credential-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.686157 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.686085    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-token\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.686157 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.686150    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-workload-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.686255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.686173    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-data\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.686380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.686367    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhspf\" (UniqueName: \"kubernetes.io/projected/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-kube-api-access-vhspf\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.686438 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.686403    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-envoy\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.787621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.787584    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-workload-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.787621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.787629    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-data\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.787975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.787954    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhspf\" (UniqueName: \"kubernetes.io/projected/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-kube-api-access-vhspf\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.788031 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.788013    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-envoy\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.788097 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.788069    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-workload-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.788148 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.788101    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istiod-ca-cert\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.788148 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.788119    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-data\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.788148 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.788130    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-podinfo\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.788394 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.788367    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-workload-certs\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.788582 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.788457    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-credential-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.788582 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.788564    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-token\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.788794 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.788685    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-workload-certs\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.788960 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.788921    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istiod-ca-cert\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.789054 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.788968    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-credential-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.791413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.791386    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-envoy\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.791882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.791862    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-podinfo\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.816707 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.816631    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhspf\" (UniqueName: \"kubernetes.io/projected/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-kube-api-access-vhspf\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.820824 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.820771    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-token\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:39.882187 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:39.882146    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:40.225606 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:45:40.225568    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598c2f2f_ccef_4d28_a82b_b8fb8fe5d170.slice/crio-4837704cb7faafe577542cf30c03077e56949c2f2e6d0c667f82cc63845738ed WatchSource:0}: Error finding container 4837704cb7faafe577542cf30c03077e56949c2f2e6d0c667f82cc63845738ed: Status 404 returned error can't find the container with id 4837704cb7faafe577542cf30c03077e56949c2f2e6d0c667f82cc63845738ed
Oct 29 22:45:40.228455 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:40.228424    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"]
Oct 29 22:45:40.229080 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:40.229040    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:45:40.229189 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:40.229133    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:45:40.229189 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:40.229175    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:45:40.355255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:40.355209    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr" event={"ID":"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170","Type":"ContainerStarted","Data":"f741cec91e60c6fa882613f53b0a70507df8efbe074b62b2f432f35cf8b1b3d0"}
Oct 29 22:45:40.355461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:40.355283    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr" event={"ID":"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170","Type":"ContainerStarted","Data":"4837704cb7faafe577542cf30c03077e56949c2f2e6d0c667f82cc63845738ed"}
Oct 29 22:45:40.370275 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:40.370223    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:45:40.413375 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:40.413309    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr" podStartSLOduration=1.41329438 podStartE2EDuration="1.41329438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:40.409043053 +0000 UTC m=+1879.320486609" watchObservedRunningTime="2026-10-29 22:45:40.41329438 +0000 UTC m=+1879.324737933"
Oct 29 22:45:40.882946 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:40.882904    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:41.787106 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.787068    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"]
Oct 29 22:45:41.794226 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.794197    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.813723 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.813690    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"]
Oct 29 22:45:41.818966 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.818934    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fee294-openshift-default-dockercfg-jk6b9\""
Oct 29 22:45:41.875479 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.875350    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-data\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.875686 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.875520    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-credential-socket\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.875686 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.875563    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-token\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.875686 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.875660    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4wxx\" (UniqueName: \"kubernetes.io/projected/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-kube-api-access-k4wxx\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.875939 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.875764    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istiod-ca-cert\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.875939 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.875803    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-envoy\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.875939 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.875862    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-workload-certs\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.875939 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.875934    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-podinfo\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.876163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.875982    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-workload-socket\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.884597 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.884568    4559 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:45:41.885079 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.885038    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr" podUID="598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 29 22:45:41.977643 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.977595    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-credential-socket\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.977843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.977661    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-token\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.977843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.977747    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4wxx\" (UniqueName: \"kubernetes.io/projected/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-kube-api-access-k4wxx\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.977952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.977861    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istiod-ca-cert\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.977952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.977900    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-envoy\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.977952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.977938    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-workload-certs\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.978051 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.978001    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-podinfo\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.978086 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.978047    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-workload-socket\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.978086 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.978053    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-credential-socket\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.978393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.978363    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-workload-certs\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.978476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.978458    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-data\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.978516 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.978468    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-workload-socket\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.978620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.978594    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istiod-ca-cert\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.978848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.978824    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-data\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.980754 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.980727    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-envoy\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.981211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.981187    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-podinfo\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.998838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.998791    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4wxx\" (UniqueName: \"kubernetes.io/projected/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-kube-api-access-k4wxx\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:41.998988 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:41.998937    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-token\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:42.114230 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.114135    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:42.386419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.386112    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"]
Oct 29 22:45:42.399555 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.399518    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.428293 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.428259    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"]
Oct 29 22:45:42.508384 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.508344    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg" podUID="83c4474e-4311-4cc6-a79f-aa4bc4edfcb0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:45:42.565048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.564995    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702eed95-27cd-42a7-a7fb-f14bcc951474-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.565267 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.565206    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.565381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.565352    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc98s\" (UniqueName: \"kubernetes.io/projected/702eed95-27cd-42a7-a7fb-f14bcc951474-kube-api-access-bc98s\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.565640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.565617    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.565766 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.565675    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.565868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.565798    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.565960 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.565898    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.566039 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.565991    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.566435 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.566411    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.667911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.667867    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.668121 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.667925    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.668121 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.667989    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.668121 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.668043    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.668332 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.668146    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.668332 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.668246    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.668332 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.668306    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702eed95-27cd-42a7-a7fb-f14bcc951474-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.668501 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.668384    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.668501 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.668453    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc98s\" (UniqueName: \"kubernetes.io/projected/702eed95-27cd-42a7-a7fb-f14bcc951474-kube-api-access-bc98s\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.669330 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.669305    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.670262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.670174    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.670676 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.670652    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.671288 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.671265    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702eed95-27cd-42a7-a7fb-f14bcc951474-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.671498 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.671482    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.673025 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.673001    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.673212 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.673195    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.686683 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.686651    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"]
Oct 29 22:45:42.689467 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:45:42.689431    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659c8f0f_bcb4_42c8_8c99_8263dfb4b97e.slice/crio-e47d16f49b5c0644432e5c87381aebfd32dd2f8c971447569ac66e30a01e868a WatchSource:0}: Error finding container e47d16f49b5c0644432e5c87381aebfd32dd2f8c971447569ac66e30a01e868a: Status 404 returned error can't find the container with id e47d16f49b5c0644432e5c87381aebfd32dd2f8c971447569ac66e30a01e868a
Oct 29 22:45:42.690072 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.690043    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.690436 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.690415    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc98s\" (UniqueName: \"kubernetes.io/projected/702eed95-27cd-42a7-a7fb-f14bcc951474-kube-api-access-bc98s\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.692646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.692614    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:45:42.692722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.692688    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:45:42.692722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.692716    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:45:42.739965 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.732803    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:42.894895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.892336    4559 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:45:42.894895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:42.892400    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr" podUID="598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 29 22:45:43.108447 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:43.108411    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"]
Oct 29 22:45:43.109047 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:45:43.109014    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702eed95_27cd_42a7_a7fb_f14bcc951474.slice/crio-a90e18bc5b9f418eed5a1b23e0ebc7629a957d0dd4d79474e26d5ce45a7511c8 WatchSource:0}: Error finding container a90e18bc5b9f418eed5a1b23e0ebc7629a957d0dd4d79474e26d5ce45a7511c8: Status 404 returned error can't find the container with id a90e18bc5b9f418eed5a1b23e0ebc7629a957d0dd4d79474e26d5ce45a7511c8
Oct 29 22:45:43.112759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:43.112718    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:45:43.112892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:43.112788    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:45:43.112892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:43.112862    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:45:43.423697 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:43.423599    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx" event={"ID":"702eed95-27cd-42a7-a7fb-f14bcc951474","Type":"ContainerStarted","Data":"43bc27af0f5395c34baf9822707c89f77e84d0f7787e517d91b93ece94194a51"}
Oct 29 22:45:43.424020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:43.423991    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx" event={"ID":"702eed95-27cd-42a7-a7fb-f14bcc951474","Type":"ContainerStarted","Data":"a90e18bc5b9f418eed5a1b23e0ebc7629a957d0dd4d79474e26d5ce45a7511c8"}
Oct 29 22:45:43.425448 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:43.425412    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs" event={"ID":"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e","Type":"ContainerStarted","Data":"38e68ab6cfca49a466d8fe8fa4f314259d91a1e748de9ec37b5f68f5a528f513"}
Oct 29 22:45:43.425572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:43.425461    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs" event={"ID":"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e","Type":"ContainerStarted","Data":"e47d16f49b5c0644432e5c87381aebfd32dd2f8c971447569ac66e30a01e868a"}
Oct 29 22:45:43.485284 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:43.485210    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx" podStartSLOduration=1.485188974 podStartE2EDuration="1.485188974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:43.469329719 +0000 UTC m=+1882.380773285" watchObservedRunningTime="2026-10-29 22:45:43.485188974 +0000 UTC m=+1882.396632529"
Oct 29 22:45:43.524275 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:43.524223    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs" podStartSLOduration=2.52420809 podStartE2EDuration="2.52420809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:43.516349242 +0000 UTC m=+1882.427792822" watchObservedRunningTime="2026-10-29 22:45:43.52420809 +0000 UTC m=+1882.435651644"
Oct 29 22:45:43.773900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:43.773342    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:43.820327 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:43.820284    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:45:43.888853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:43.886447    4559 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:45:43.888853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:43.886508    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr" podUID="598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 29 22:45:44.181839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:44.177923    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:44.181839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:44.178062    4559 patch_prober.go:29] interesting pod/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 29 22:45:44.181839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:44.178106    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs" podUID="659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 29 22:45:44.756836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:44.756715    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:45:44.756836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:44.756780    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx" podUID="702eed95-27cd-42a7-a7fb-f14bcc951474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 22:45:44.887355 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:44.887311    4559 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:45:44.887671 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:44.887639    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr" podUID="598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 29 22:45:45.212902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:45.143209    4559 patch_prober.go:29] interesting pod/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 29 22:45:45.212902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:45.143278    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs" podUID="659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 29 22:45:45.751263 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:45.751223    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:45:45.751590 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:45.751555    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx" podUID="702eed95-27cd-42a7-a7fb-f14bcc951474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 22:45:45.889841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:45.888294    4559 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:45:45.889841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:45.888362    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr" podUID="598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 29 22:45:46.115727 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:46.115624    4559 patch_prober.go:29] interesting pod/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 29 22:45:46.138917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:46.138871    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs" podUID="659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 29 22:45:46.753837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:46.753228    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:45:46.753837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:46.753292    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx" podUID="702eed95-27cd-42a7-a7fb-f14bcc951474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 22:45:46.883267 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:46.883225    4559 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:45:46.883581 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:46.883547    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr" podUID="598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 29 22:45:47.115001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:47.114907    4559 patch_prober.go:29] interesting pod/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 29 22:45:47.115001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:47.114965    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs" podUID="659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 29 22:45:47.755027 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:47.754991    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:45:47.755659 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:47.755622    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx" podUID="702eed95-27cd-42a7-a7fb-f14bcc951474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 22:45:47.891153 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:47.891118    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:47.891694 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:47.891669    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:47.892449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:47.892416    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:45:48.128480 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:48.127176    4559 patch_prober.go:29] interesting pod/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 29 22:45:48.128480 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:48.127236    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs" podUID="659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 29 22:45:48.755837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:48.755237    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:45:48.755837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:48.755305    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx" podUID="702eed95-27cd-42a7-a7fb-f14bcc951474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 22:45:49.114856 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:49.114755    4559 patch_prober.go:29] interesting pod/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 29 22:45:49.114856 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:49.114821    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs" podUID="659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 29 22:45:49.750294 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:49.750249    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:45:49.750599 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:49.750562    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx" podUID="702eed95-27cd-42a7-a7fb-f14bcc951474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 22:45:50.120602 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:50.120501    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:50.121172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:50.121047    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:50.121769 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:50.121734    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:45:50.754071 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:50.754023    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:50.754617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:50.754589    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:50.755482 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:50.755457    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:45:52.507603 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:52.507558    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg" podUID="83c4474e-4311-4cc6-a79f-aa4bc4edfcb0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:45:55.209699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:55.209665    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"
Oct 29 22:45:55.306130 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:55.306090    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c4474e-4311-4cc6-a79f-aa4bc4edfcb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c4474e-4311-4cc6-a79f-aa4bc4edfcb0-plugins-config-volume\") pod \"83c4474e-4311-4cc6-a79f-aa4bc4edfcb0\" (UID: \"83c4474e-4311-4cc6-a79f-aa4bc4edfcb0\") "
Oct 29 22:45:55.306348 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:55.306162    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c4474e-4311-4cc6-a79f-aa4bc4edfcb0-kube-api-access-pj5pw\" (UniqueName: \"kubernetes.io/projected/83c4474e-4311-4cc6-a79f-aa4bc4edfcb0-kube-api-access-pj5pw\") pod \"83c4474e-4311-4cc6-a79f-aa4bc4edfcb0\" (UID: \"83c4474e-4311-4cc6-a79f-aa4bc4edfcb0\") "
Oct 29 22:45:55.306481 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:55.306452    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c4474e-4311-4cc6-a79f-aa4bc4edfcb0-plugins-config-volume" pod "83c4474e-4311-4cc6-a79f-aa4bc4edfcb0" (UID: "83c4474e-4311-4cc6-a79f-aa4bc4edfcb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:55.307311 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:55.307282    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c4474e-4311-4cc6-a79f-aa4bc4edfcb0-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:55.309195 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:55.309169    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c4474e-4311-4cc6-a79f-aa4bc4edfcb0-kube-api-access-pj5pw" pod "83c4474e-4311-4cc6-a79f-aa4bc4edfcb0" (UID: "83c4474e-4311-4cc6-a79f-aa4bc4edfcb0"). InnerVolumeSpecName "kube-api-access-pj5pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:55.408842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:55.408789    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj5pw\" (UniqueName: \"kubernetes.io/projected/83c4474e-4311-4cc6-a79f-aa4bc4edfcb0-kube-api-access-pj5pw\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:45:55.521426 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:55.521327    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"]
Oct 29 22:45:55.735846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:55.732776    4559 generic.go:356] "Generic (PLEG): container finished" podID="83c4474e-4311-4cc6-a79f-aa4bc4edfcb0" containerID="955dac93fb7a24f9d879fea41427f2aef7e1fbd8ee1b17ad7a8fa70f2af9eb89" exitCode=0
Oct 29 22:45:55.735846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:55.733000    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg" event={"ID":"83c4474e-4311-4cc6-a79f-aa4bc4edfcb0","Type":"ContainerDied","Data":"955dac93fb7a24f9d879fea41427f2aef7e1fbd8ee1b17ad7a8fa70f2af9eb89"}
Oct 29 22:45:55.735846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:55.733063    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg" event={"ID":"83c4474e-4311-4cc6-a79f-aa4bc4edfcb0","Type":"ContainerDied","Data":"2cc3d523f8b8322f8c05f7035eba05083a9536d205505a7bb8d7a76a2e6c8ce2"}
Oct 29 22:45:55.735846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:55.733103    4559 scope.go:122] "RemoveContainer" containerID="955dac93fb7a24f9d879fea41427f2aef7e1fbd8ee1b17ad7a8fa70f2af9eb89"
Oct 29 22:45:55.735846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:55.733491    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"
Oct 29 22:45:55.828015 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:55.825998    4559 scope.go:122] "RemoveContainer" containerID="955dac93fb7a24f9d879fea41427f2aef7e1fbd8ee1b17ad7a8fa70f2af9eb89"
Oct 29 22:45:55.831864 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:45:55.830974    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"955dac93fb7a24f9d879fea41427f2aef7e1fbd8ee1b17ad7a8fa70f2af9eb89\": container with ID starting with 955dac93fb7a24f9d879fea41427f2aef7e1fbd8ee1b17ad7a8fa70f2af9eb89 not found: ID does not exist" containerID="955dac93fb7a24f9d879fea41427f2aef7e1fbd8ee1b17ad7a8fa70f2af9eb89"
Oct 29 22:45:55.831864 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:55.831024    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"955dac93fb7a24f9d879fea41427f2aef7e1fbd8ee1b17ad7a8fa70f2af9eb89"} err="failed to get container status \"955dac93fb7a24f9d879fea41427f2aef7e1fbd8ee1b17ad7a8fa70f2af9eb89\": rpc error: code = NotFound desc = could not find container \"955dac93fb7a24f9d879fea41427f2aef7e1fbd8ee1b17ad7a8fa70f2af9eb89\": container with ID starting with 955dac93fb7a24f9d879fea41427f2aef7e1fbd8ee1b17ad7a8fa70f2af9eb89 not found: ID does not exist"
Oct 29 22:45:55.866606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:55.859655    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"]
Oct 29 22:45:55.879899 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:55.879865    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c296a3-778459576c-5mnqg"]
Oct 29 22:45:57.231308 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:45:57.231277    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c4474e-4311-4cc6-a79f-aa4bc4edfcb0" path="/var/lib/kubelet/pods/83c4474e-4311-4cc6-a79f-aa4bc4edfcb0/volumes"
Oct 29 22:46:04.872731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:04.872692    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"]
Oct 29 22:46:04.900601 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:04.900084    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c296a3-8lxfz"]
Oct 29 22:46:05.230430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:05.230393    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6324171d-890b-481b-bf15-29dec862d313" path="/var/lib/kubelet/pods/6324171d-890b-481b-bf15-29dec862d313/volumes"
Oct 29 22:46:10.550009 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.549931    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4" podUID="e44215a6-17c6-48e9-b4d6-123b1ef1ac12" containerName="main" containerID="cri-o://e3b338131e0b9db5be5ae269715b167dd318f713e8b24d3c669fadee0e564705" gracePeriod=45
Oct 29 22:46:10.691873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.691822    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4" podUID="e44215a6-17c6-48e9-b4d6-123b1ef1ac12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.144:8000/ready\": dial tcp 10.134.0.144:8000: connect: connection refused"
Oct 29 22:46:10.871916 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.871888    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:46:10.943064 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.943019    4559 generic.go:356] "Generic (PLEG): container finished" podID="e44215a6-17c6-48e9-b4d6-123b1ef1ac12" containerID="e3b338131e0b9db5be5ae269715b167dd318f713e8b24d3c669fadee0e564705" exitCode=0
Oct 29 22:46:10.943243 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.943183    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4" event={"ID":"e44215a6-17c6-48e9-b4d6-123b1ef1ac12","Type":"ContainerDied","Data":"e3b338131e0b9db5be5ae269715b167dd318f713e8b24d3c669fadee0e564705"}
Oct 29 22:46:10.943243 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.943225    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4" event={"ID":"e44215a6-17c6-48e9-b4d6-123b1ef1ac12","Type":"ContainerDied","Data":"6596d5122946fb15231892a69206708cd05d98359edf1adace52d3c5988a54b0"}
Oct 29 22:46:10.943413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.943249    4559 scope.go:122] "RemoveContainer" containerID="e3b338131e0b9db5be5ae269715b167dd318f713e8b24d3c669fadee0e564705"
Oct 29 22:46:10.943475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.943427    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"
Oct 29 22:46:10.959030 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.957987    4559 scope.go:122] "RemoveContainer" containerID="e3b338131e0b9db5be5ae269715b167dd318f713e8b24d3c669fadee0e564705"
Oct 29 22:46:10.959030 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:46:10.958440    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b338131e0b9db5be5ae269715b167dd318f713e8b24d3c669fadee0e564705\": container with ID starting with e3b338131e0b9db5be5ae269715b167dd318f713e8b24d3c669fadee0e564705 not found: ID does not exist" containerID="e3b338131e0b9db5be5ae269715b167dd318f713e8b24d3c669fadee0e564705"
Oct 29 22:46:10.959030 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.958525    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b338131e0b9db5be5ae269715b167dd318f713e8b24d3c669fadee0e564705"} err="failed to get container status \"e3b338131e0b9db5be5ae269715b167dd318f713e8b24d3c669fadee0e564705\": rpc error: code = NotFound desc = could not find container \"e3b338131e0b9db5be5ae269715b167dd318f713e8b24d3c669fadee0e564705\": container with ID starting with e3b338131e0b9db5be5ae269715b167dd318f713e8b24d3c669fadee0e564705 not found: ID does not exist"
Oct 29 22:46:10.961698 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.961653    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-tls-certs\" (UniqueName: \"kubernetes.io/secret/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-tls-certs\") pod \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") "
Oct 29 22:46:10.961886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.961724    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-kube-api-access-9s7p4\" (UniqueName: \"kubernetes.io/projected/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-kube-api-access-9s7p4\") pod \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") "
Oct 29 22:46:10.961995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.961909    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-tmp-dir\") pod \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") "
Oct 29 22:46:10.962175 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.962100    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-model-cache\") pod \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") "
Oct 29 22:46:10.962175 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.962136    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-home\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-home\") pod \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") "
Oct 29 22:46:10.962336 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.962247    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-dshm\") pod \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\" (UID: \"e44215a6-17c6-48e9-b4d6-123b1ef1ac12\") "
Oct 29 22:46:10.963948 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.963919    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-home" pod "e44215a6-17c6-48e9-b4d6-123b1ef1ac12" (UID: "e44215a6-17c6-48e9-b4d6-123b1ef1ac12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:10.964216 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.964187    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-model-cache" pod "e44215a6-17c6-48e9-b4d6-123b1ef1ac12" (UID: "e44215a6-17c6-48e9-b4d6-123b1ef1ac12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:10.964532 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.964508    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-tmp-dir" pod "e44215a6-17c6-48e9-b4d6-123b1ef1ac12" (UID: "e44215a6-17c6-48e9-b4d6-123b1ef1ac12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:10.980885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.967731    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-kube-api-access-9s7p4" pod "e44215a6-17c6-48e9-b4d6-123b1ef1ac12" (UID: "e44215a6-17c6-48e9-b4d6-123b1ef1ac12"). InnerVolumeSpecName "kube-api-access-9s7p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:10.980885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.968065    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-tls-certs" pod "e44215a6-17c6-48e9-b4d6-123b1ef1ac12" (UID: "e44215a6-17c6-48e9-b4d6-123b1ef1ac12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:46:10.980885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:10.968161    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-dshm" pod "e44215a6-17c6-48e9-b4d6-123b1ef1ac12" (UID: "e44215a6-17c6-48e9-b4d6-123b1ef1ac12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:11.064640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:11.064599    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:11.064640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:11.064632    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:11.064640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:11.064640    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:11.064640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:11.064648    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:11.064940 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:11.064657    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s7p4\" (UniqueName: \"kubernetes.io/projected/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-kube-api-access-9s7p4\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:11.064940 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:11.064665    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44215a6-17c6-48e9-b4d6-123b1ef1ac12-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:11.289231 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:11.289186    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"]
Oct 29 22:46:11.293368 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:11.293337    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c1750b/test-model-c1750b-kserve-7969d8bf9f-dj2m4"]
Oct 29 22:46:12.597965 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:12.597932    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"]
Oct 29 22:46:12.598583 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:12.598132    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4" podUID="65fe44fa-7821-4581-9607-05d919e94273" containerName="istio-proxy" containerID="cri-o://ee6c01931c61f72ec9f143422e52652960e451c1a0ac352a7a7665fca26f4e0a" gracePeriod=30
Oct 29 22:46:13.230038 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:13.230004    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e44215a6-17c6-48e9-b4d6-123b1ef1ac12" path="/var/lib/kubelet/pods/e44215a6-17c6-48e9-b4d6-123b1ef1ac12/volumes"
Oct 29 22:46:13.697008 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:46:13.696972    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:46:17.984426 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:17.984391    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:46:18.000251 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.000215    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-istio-envoy\") pod \"65fe44fa-7821-4581-9607-05d919e94273\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") "
Oct 29 22:46:18.000447 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.000330    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-credential-socket\") pod \"65fe44fa-7821-4581-9607-05d919e94273\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") "
Oct 29 22:46:18.000447 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.000427    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65fe44fa-7821-4581-9607-05d919e94273-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65fe44fa-7821-4581-9607-05d919e94273-istiod-ca-cert\") pod \"65fe44fa-7821-4581-9607-05d919e94273\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") "
Oct 29 22:46:18.000578 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.000484    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65fe44fa-7821-4581-9607-05d919e94273-istio-token\" (UniqueName: \"kubernetes.io/projected/65fe44fa-7821-4581-9607-05d919e94273-istio-token\") pod \"65fe44fa-7821-4581-9607-05d919e94273\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") "
Oct 29 22:46:18.000578 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.000546    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-istio-data\") pod \"65fe44fa-7821-4581-9607-05d919e94273\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") "
Oct 29 22:46:18.000707 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.000637    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65fe44fa-7821-4581-9607-05d919e94273-kube-api-access-wrtfw\" (UniqueName: \"kubernetes.io/projected/65fe44fa-7821-4581-9607-05d919e94273-kube-api-access-wrtfw\") pod \"65fe44fa-7821-4581-9607-05d919e94273\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") "
Oct 29 22:46:18.000856 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.000790    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-workload-certs\") pod \"65fe44fa-7821-4581-9607-05d919e94273\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") "
Oct 29 22:46:18.000947 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.000907    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-workload-socket\") pod \"65fe44fa-7821-4581-9607-05d919e94273\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") "
Oct 29 22:46:18.001013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.000946    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65fe44fa-7821-4581-9607-05d919e94273-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65fe44fa-7821-4581-9607-05d919e94273-istio-podinfo\") pod \"65fe44fa-7821-4581-9607-05d919e94273\" (UID: \"65fe44fa-7821-4581-9607-05d919e94273\") "
Oct 29 22:46:18.004594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.004560    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65fe44fa-7821-4581-9607-05d919e94273-istio-podinfo" pod "65fe44fa-7821-4581-9607-05d919e94273" (UID: "65fe44fa-7821-4581-9607-05d919e94273"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:46:18.005064 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.005038    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-istio-data" pod "65fe44fa-7821-4581-9607-05d919e94273" (UID: "65fe44fa-7821-4581-9607-05d919e94273"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:18.005450 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.005186    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-credential-socket" pod "65fe44fa-7821-4581-9607-05d919e94273" (UID: "65fe44fa-7821-4581-9607-05d919e94273"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:18.005550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.005496    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-workload-certs" pod "65fe44fa-7821-4581-9607-05d919e94273" (UID: "65fe44fa-7821-4581-9607-05d919e94273"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:18.005798 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.005775    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65fe44fa-7821-4581-9607-05d919e94273-istiod-ca-cert" pod "65fe44fa-7821-4581-9607-05d919e94273" (UID: "65fe44fa-7821-4581-9607-05d919e94273"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:46:18.027612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.006445    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-workload-socket" pod "65fe44fa-7821-4581-9607-05d919e94273" (UID: "65fe44fa-7821-4581-9607-05d919e94273"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:18.028425 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.028386    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65fe44fa-7821-4581-9607-05d919e94273-istio-token" pod "65fe44fa-7821-4581-9607-05d919e94273" (UID: "65fe44fa-7821-4581-9607-05d919e94273"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:18.031281 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.031246    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65fe44fa-7821-4581-9607-05d919e94273-kube-api-access-wrtfw" pod "65fe44fa-7821-4581-9607-05d919e94273" (UID: "65fe44fa-7821-4581-9607-05d919e94273"). InnerVolumeSpecName "kube-api-access-wrtfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:18.032924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.032892    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-istio-envoy" pod "65fe44fa-7821-4581-9607-05d919e94273" (UID: "65fe44fa-7821-4581-9607-05d919e94273"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:18.103982 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.103940    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrtfw\" (UniqueName: \"kubernetes.io/projected/65fe44fa-7821-4581-9607-05d919e94273-kube-api-access-wrtfw\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.103982 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.103980    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.104211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.103995    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.104211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.104007    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65fe44fa-7821-4581-9607-05d919e94273-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.104211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.104019    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.104211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.104031    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.104211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.104045    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65fe44fa-7821-4581-9607-05d919e94273-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.104211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.104057    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65fe44fa-7821-4581-9607-05d919e94273-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.104211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.104069    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65fe44fa-7821-4581-9607-05d919e94273-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.504390 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.504355    4559 generic.go:356] "Generic (PLEG): container finished" podID="65fe44fa-7821-4581-9607-05d919e94273" containerID="ee6c01931c61f72ec9f143422e52652960e451c1a0ac352a7a7665fca26f4e0a" exitCode=0
Oct 29 22:46:18.504607 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.504418    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"
Oct 29 22:46:18.504607 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.504435    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4" event={"ID":"65fe44fa-7821-4581-9607-05d919e94273","Type":"ContainerDied","Data":"ee6c01931c61f72ec9f143422e52652960e451c1a0ac352a7a7665fca26f4e0a"}
Oct 29 22:46:18.504607 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.504493    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4" event={"ID":"65fe44fa-7821-4581-9607-05d919e94273","Type":"ContainerDied","Data":"d5d0c4082608562160042b2491e7f408410bdf1a4e73c47eccea8efff323a076"}
Oct 29 22:46:18.504607 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.504525    4559 scope.go:122] "RemoveContainer" containerID="ee6c01931c61f72ec9f143422e52652960e451c1a0ac352a7a7665fca26f4e0a"
Oct 29 22:46:18.517900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.517875    4559 scope.go:122] "RemoveContainer" containerID="8524e8f9021ce677259bd2a7be6982af38b91d4322f62867409f352f8e3de2a8"
Oct 29 22:46:18.530241 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.530211    4559 scope.go:122] "RemoveContainer" containerID="ee6c01931c61f72ec9f143422e52652960e451c1a0ac352a7a7665fca26f4e0a"
Oct 29 22:46:18.530566 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:46:18.530548    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee6c01931c61f72ec9f143422e52652960e451c1a0ac352a7a7665fca26f4e0a\": container with ID starting with ee6c01931c61f72ec9f143422e52652960e451c1a0ac352a7a7665fca26f4e0a not found: ID does not exist" containerID="ee6c01931c61f72ec9f143422e52652960e451c1a0ac352a7a7665fca26f4e0a"
Oct 29 22:46:18.530619 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.530576    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee6c01931c61f72ec9f143422e52652960e451c1a0ac352a7a7665fca26f4e0a"} err="failed to get container status \"ee6c01931c61f72ec9f143422e52652960e451c1a0ac352a7a7665fca26f4e0a\": rpc error: code = NotFound desc = could not find container \"ee6c01931c61f72ec9f143422e52652960e451c1a0ac352a7a7665fca26f4e0a\": container with ID starting with ee6c01931c61f72ec9f143422e52652960e451c1a0ac352a7a7665fca26f4e0a not found: ID does not exist"
Oct 29 22:46:18.530619 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.530607    4559 scope.go:122] "RemoveContainer" containerID="8524e8f9021ce677259bd2a7be6982af38b91d4322f62867409f352f8e3de2a8"
Oct 29 22:46:18.530943 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:46:18.530914    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8524e8f9021ce677259bd2a7be6982af38b91d4322f62867409f352f8e3de2a8\": container with ID starting with 8524e8f9021ce677259bd2a7be6982af38b91d4322f62867409f352f8e3de2a8 not found: ID does not exist" containerID="8524e8f9021ce677259bd2a7be6982af38b91d4322f62867409f352f8e3de2a8"
Oct 29 22:46:18.531019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.530953    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8524e8f9021ce677259bd2a7be6982af38b91d4322f62867409f352f8e3de2a8"} err="failed to get container status \"8524e8f9021ce677259bd2a7be6982af38b91d4322f62867409f352f8e3de2a8\": rpc error: code = NotFound desc = could not find container \"8524e8f9021ce677259bd2a7be6982af38b91d4322f62867409f352f8e3de2a8\": container with ID starting with 8524e8f9021ce677259bd2a7be6982af38b91d4322f62867409f352f8e3de2a8 not found: ID does not exist"
Oct 29 22:46:18.563403 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.563358    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"]
Oct 29 22:46:18.573150 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:18.573114    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-fdkc4"]
Oct 29 22:46:19.243837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:19.243278    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65fe44fa-7821-4581-9607-05d919e94273" path="/var/lib/kubelet/pods/65fe44fa-7821-4581-9607-05d919e94273/volumes"
Oct 29 22:46:20.309927 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.309895    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"]
Oct 29 22:46:20.310768 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.310751    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e44215a6-17c6-48e9-b4d6-123b1ef1ac12" containerName="main"
Oct 29 22:46:20.310840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.310769    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44215a6-17c6-48e9-b4d6-123b1ef1ac12" containerName="main"
Oct 29 22:46:20.310840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.310791    4559 cpu_manager.go:414] "containerMap: removing container" podUID="83c4474e-4311-4cc6-a79f-aa4bc4edfcb0" containerName="payload-pre-processing"
Oct 29 22:46:20.310840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.310796    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c4474e-4311-4cc6-a79f-aa4bc4edfcb0" containerName="payload-pre-processing"
Oct 29 22:46:20.310840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.310839    4559 cpu_manager.go:414] "containerMap: removing container" podUID="65fe44fa-7821-4581-9607-05d919e94273" containerName="istio-proxy"
Oct 29 22:46:20.310987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.310845    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65fe44fa-7821-4581-9607-05d919e94273" containerName="istio-proxy"
Oct 29 22:46:20.310987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.310859    4559 cpu_manager.go:414] "containerMap: removing container" podUID="65fe44fa-7821-4581-9607-05d919e94273" containerName="istio-proxy"
Oct 29 22:46:20.310987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.310864    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65fe44fa-7821-4581-9607-05d919e94273" containerName="istio-proxy"
Oct 29 22:46:20.311096 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.310987    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44215a6-17c6-48e9-b4d6-123b1ef1ac12" containerName="main"
Oct 29 22:46:20.311096 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.311000    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c4474e-4311-4cc6-a79f-aa4bc4edfcb0" containerName="payload-pre-processing"
Oct 29 22:46:20.311096 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.311009    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="65fe44fa-7821-4581-9607-05d919e94273" containerName="istio-proxy"
Oct 29 22:46:20.311096 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.311015    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="65fe44fa-7821-4581-9607-05d919e94273" containerName="istio-proxy"
Oct 29 22:46:20.316353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.316329    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"
Oct 29 22:46:20.334216 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.334183    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-af260f\""
Oct 29 22:46:20.336023 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.335596    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"]
Oct 29 22:46:20.341752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.341710    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-af260f-dockercfg-k4hb6\""
Oct 29 22:46:20.344761 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.344732    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5q6c\" (UniqueName: \"kubernetes.io/projected/a5f30faf-8940-4ded-b40e-7897f7e4a22b-kube-api-access-s5q6c\") pod \"payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq\" (UID: \"a5f30faf-8940-4ded-b40e-7897f7e4a22b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"
Oct 29 22:46:20.344964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.344791    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f30faf-8940-4ded-b40e-7897f7e4a22b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq\" (UID: \"a5f30faf-8940-4ded-b40e-7897f7e4a22b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"
Oct 29 22:46:20.485839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.471396    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5q6c\" (UniqueName: \"kubernetes.io/projected/a5f30faf-8940-4ded-b40e-7897f7e4a22b-kube-api-access-s5q6c\") pod \"payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq\" (UID: \"a5f30faf-8940-4ded-b40e-7897f7e4a22b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"
Oct 29 22:46:20.485839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.471489    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f30faf-8940-4ded-b40e-7897f7e4a22b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq\" (UID: \"a5f30faf-8940-4ded-b40e-7897f7e4a22b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"
Oct 29 22:46:20.485839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.472613    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f30faf-8940-4ded-b40e-7897f7e4a22b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq\" (UID: \"a5f30faf-8940-4ded-b40e-7897f7e4a22b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"
Oct 29 22:46:20.743943 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.743908    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"]
Oct 29 22:46:20.754671 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.754642    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"]
Oct 29 22:46:20.755024 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.755006    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"
Oct 29 22:46:20.776374 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.776343    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5q6c\" (UniqueName: \"kubernetes.io/projected/a5f30faf-8940-4ded-b40e-7897f7e4a22b-kube-api-access-s5q6c\") pod \"payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq\" (UID: \"a5f30faf-8940-4ded-b40e-7897f7e4a22b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"
Oct 29 22:46:20.851150 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.851105    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e75cc1b-5e4f-403d-8501-a2dccd8e99a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz\" (UID: \"6e75cc1b-5e4f-403d-8501-a2dccd8e99a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"
Oct 29 22:46:20.851373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.851300    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfbj5\" (UniqueName: \"kubernetes.io/projected/6e75cc1b-5e4f-403d-8501-a2dccd8e99a1-kube-api-access-vfbj5\") pod \"payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz\" (UID: \"6e75cc1b-5e4f-403d-8501-a2dccd8e99a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"
Oct 29 22:46:20.949835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.943264    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"
Oct 29 22:46:20.955849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.953148    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfbj5\" (UniqueName: \"kubernetes.io/projected/6e75cc1b-5e4f-403d-8501-a2dccd8e99a1-kube-api-access-vfbj5\") pod \"payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz\" (UID: \"6e75cc1b-5e4f-403d-8501-a2dccd8e99a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"
Oct 29 22:46:20.955849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.953475    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e75cc1b-5e4f-403d-8501-a2dccd8e99a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz\" (UID: \"6e75cc1b-5e4f-403d-8501-a2dccd8e99a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"
Oct 29 22:46:20.971660 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.971622    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e75cc1b-5e4f-403d-8501-a2dccd8e99a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz\" (UID: \"6e75cc1b-5e4f-403d-8501-a2dccd8e99a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"
Oct 29 22:46:20.994195 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:20.994110    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfbj5\" (UniqueName: \"kubernetes.io/projected/6e75cc1b-5e4f-403d-8501-a2dccd8e99a1-kube-api-access-vfbj5\") pod \"payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz\" (UID: \"6e75cc1b-5e4f-403d-8501-a2dccd8e99a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"
Oct 29 22:46:21.098490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:21.098449    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"
Oct 29 22:46:21.314349 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:46:21.314314    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f30faf_8940_4ded_b40e_7897f7e4a22b.slice/crio-107de6bf6032c646be9e81ced8f1247982a5fa4e5bde61b1a8350797a7a4a8c3 WatchSource:0}: Error finding container 107de6bf6032c646be9e81ced8f1247982a5fa4e5bde61b1a8350797a7a4a8c3: Status 404 returned error can't find the container with id 107de6bf6032c646be9e81ced8f1247982a5fa4e5bde61b1a8350797a7a4a8c3
Oct 29 22:46:21.315641 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:21.315611    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"]
Oct 29 22:46:21.430020 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:46:21.429982    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e75cc1b_5e4f_403d_8501_a2dccd8e99a1.slice/crio-1f271c01c771c990ce58515dde70b5493125bd5958aaee9176efb947ee76b4e8 WatchSource:0}: Error finding container 1f271c01c771c990ce58515dde70b5493125bd5958aaee9176efb947ee76b4e8: Status 404 returned error can't find the container with id 1f271c01c771c990ce58515dde70b5493125bd5958aaee9176efb947ee76b4e8
Oct 29 22:46:21.431014 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:21.430982    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"]
Oct 29 22:46:21.784094 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:21.784043    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz" event={"ID":"6e75cc1b-5e4f-403d-8501-a2dccd8e99a1","Type":"ContainerStarted","Data":"e20204eed6f2e3f319b3c78934d90b7aacc8772c3463fad7962b07328ae03e2b"}
Oct 29 22:46:21.784291 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:21.784111    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz" event={"ID":"6e75cc1b-5e4f-403d-8501-a2dccd8e99a1","Type":"ContainerStarted","Data":"1f271c01c771c990ce58515dde70b5493125bd5958aaee9176efb947ee76b4e8"}
Oct 29 22:46:21.784364 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:21.784307    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"
Oct 29 22:46:21.786279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:21.786246    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq" event={"ID":"a5f30faf-8940-4ded-b40e-7897f7e4a22b","Type":"ContainerStarted","Data":"d874ee6316165d3f7e06a5d1283e891535a44082c46bb9334a2e64a793044e41"}
Oct 29 22:46:21.786430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:21.786296    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq" event={"ID":"a5f30faf-8940-4ded-b40e-7897f7e4a22b","Type":"ContainerStarted","Data":"107de6bf6032c646be9e81ced8f1247982a5fa4e5bde61b1a8350797a7a4a8c3"}
Oct 29 22:46:21.786430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:21.786340    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"
Oct 29 22:46:21.830569 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:21.830511    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz" podStartSLOduration=1.830495655 podStartE2EDuration="1.830495655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:21.826202357 +0000 UTC m=+1920.737645911" watchObservedRunningTime="2026-10-29 22:46:21.830495655 +0000 UTC m=+1920.741939209"
Oct 29 22:46:21.878151 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:21.878084    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq" podStartSLOduration=1.878065389 podStartE2EDuration="1.878065389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:21.874224495 +0000 UTC m=+1920.785668050" watchObservedRunningTime="2026-10-29 22:46:21.878065389 +0000 UTC m=+1920.789508945"
Oct 29 22:46:32.799671 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:32.799470    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"
Oct 29 22:46:32.800106 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:32.799885    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"
Oct 29 22:46:38.986308 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:38.985705    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"]
Oct 29 22:46:38.986308 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:38.985985    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8" podUID="ca5a7c39-6931-4960-932c-910ad8d3c53a" containerName="istio-proxy" containerID="cri-o://5575c0b20a085faef27efaac24aa1d614172f76c698dd1f021cfa78f44b8db78" gracePeriod=30
Oct 29 22:46:43.697593 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:46:43.697542    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:46:44.312654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.312625    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:46:44.393121 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.393087    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-workload-certs\") pod \"ca5a7c39-6931-4960-932c-910ad8d3c53a\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") "
Oct 29 22:46:44.393359 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.393338    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-credential-socket\") pod \"ca5a7c39-6931-4960-932c-910ad8d3c53a\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") "
Oct 29 22:46:44.393451 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.393378    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-envoy\") pod \"ca5a7c39-6931-4960-932c-910ad8d3c53a\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") "
Oct 29 22:46:44.393451 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.393426    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-podinfo\") pod \"ca5a7c39-6931-4960-932c-910ad8d3c53a\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") "
Oct 29 22:46:44.393451 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.393428    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-workload-certs" pod "ca5a7c39-6931-4960-932c-910ad8d3c53a" (UID: "ca5a7c39-6931-4960-932c-910ad8d3c53a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:44.393609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.393466    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5a7c39-6931-4960-932c-910ad8d3c53a-kube-api-access-72l99\" (UniqueName: \"kubernetes.io/projected/ca5a7c39-6931-4960-932c-910ad8d3c53a-kube-api-access-72l99\") pod \"ca5a7c39-6931-4960-932c-910ad8d3c53a\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") "
Oct 29 22:46:44.393609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.393507    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-workload-socket\") pod \"ca5a7c39-6931-4960-932c-910ad8d3c53a\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") "
Oct 29 22:46:44.393609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.393554    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-data\") pod \"ca5a7c39-6931-4960-932c-910ad8d3c53a\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") "
Oct 29 22:46:44.393776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.393759    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-token\" (UniqueName: \"kubernetes.io/projected/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-token\") pod \"ca5a7c39-6931-4960-932c-910ad8d3c53a\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") "
Oct 29 22:46:44.393966 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.393948    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5a7c39-6931-4960-932c-910ad8d3c53a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5a7c39-6931-4960-932c-910ad8d3c53a-istiod-ca-cert\") pod \"ca5a7c39-6931-4960-932c-910ad8d3c53a\" (UID: \"ca5a7c39-6931-4960-932c-910ad8d3c53a\") "
Oct 29 22:46:44.394278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.394013    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-workload-socket" pod "ca5a7c39-6931-4960-932c-910ad8d3c53a" (UID: "ca5a7c39-6931-4960-932c-910ad8d3c53a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:44.394434 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.394333    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5a7c39-6931-4960-932c-910ad8d3c53a-istiod-ca-cert" pod "ca5a7c39-6931-4960-932c-910ad8d3c53a" (UID: "ca5a7c39-6931-4960-932c-910ad8d3c53a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:46:44.394556 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.394510    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-data" pod "ca5a7c39-6931-4960-932c-910ad8d3c53a" (UID: "ca5a7c39-6931-4960-932c-910ad8d3c53a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:44.395108 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.395079    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5a7c39-6931-4960-932c-910ad8d3c53a-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:44.395195 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.395116    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:44.395195 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.395130    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:44.395195 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.395141    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:44.395314 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.395234    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-credential-socket" pod "ca5a7c39-6931-4960-932c-910ad8d3c53a" (UID: "ca5a7c39-6931-4960-932c-910ad8d3c53a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:44.396872 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.396845    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-token" pod "ca5a7c39-6931-4960-932c-910ad8d3c53a" (UID: "ca5a7c39-6931-4960-932c-910ad8d3c53a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:44.397187 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.397156    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5a7c39-6931-4960-932c-910ad8d3c53a-kube-api-access-72l99" pod "ca5a7c39-6931-4960-932c-910ad8d3c53a" (UID: "ca5a7c39-6931-4960-932c-910ad8d3c53a"). InnerVolumeSpecName "kube-api-access-72l99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:44.398676 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.398657    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-envoy" pod "ca5a7c39-6931-4960-932c-910ad8d3c53a" (UID: "ca5a7c39-6931-4960-932c-910ad8d3c53a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:44.398977 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.398958    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-podinfo" pod "ca5a7c39-6931-4960-932c-910ad8d3c53a" (UID: "ca5a7c39-6931-4960-932c-910ad8d3c53a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:46:44.501833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.497804    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:44.501833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.497859    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:44.501833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.497876    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:44.501833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.497888    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72l99\" (UniqueName: \"kubernetes.io/projected/ca5a7c39-6931-4960-932c-910ad8d3c53a-kube-api-access-72l99\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:44.501833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:44.497901    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5a7c39-6931-4960-932c-910ad8d3c53a-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:45.053690 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:45.053657    4559 generic.go:356] "Generic (PLEG): container finished" podID="ca5a7c39-6931-4960-932c-910ad8d3c53a" containerID="5575c0b20a085faef27efaac24aa1d614172f76c698dd1f021cfa78f44b8db78" exitCode=0
Oct 29 22:46:45.054154 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:45.053713    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"
Oct 29 22:46:45.054154 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:45.053745    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8" event={"ID":"ca5a7c39-6931-4960-932c-910ad8d3c53a","Type":"ContainerDied","Data":"5575c0b20a085faef27efaac24aa1d614172f76c698dd1f021cfa78f44b8db78"}
Oct 29 22:46:45.054154 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:45.053800    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8" event={"ID":"ca5a7c39-6931-4960-932c-910ad8d3c53a","Type":"ContainerDied","Data":"9e9f369a7c00f6f0e1a8e0f40f514b54680ec2eebb808a66a7104e85fd54437b"}
Oct 29 22:46:45.054154 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:45.053839    4559 scope.go:122] "RemoveContainer" containerID="5575c0b20a085faef27efaac24aa1d614172f76c698dd1f021cfa78f44b8db78"
Oct 29 22:46:45.066566 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:45.066543    4559 scope.go:122] "RemoveContainer" containerID="5575c0b20a085faef27efaac24aa1d614172f76c698dd1f021cfa78f44b8db78"
Oct 29 22:46:45.066970 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:46:45.066944    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5575c0b20a085faef27efaac24aa1d614172f76c698dd1f021cfa78f44b8db78\": container with ID starting with 5575c0b20a085faef27efaac24aa1d614172f76c698dd1f021cfa78f44b8db78 not found: ID does not exist" containerID="5575c0b20a085faef27efaac24aa1d614172f76c698dd1f021cfa78f44b8db78"
Oct 29 22:46:45.067088 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:45.066977    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5575c0b20a085faef27efaac24aa1d614172f76c698dd1f021cfa78f44b8db78"} err="failed to get container status \"5575c0b20a085faef27efaac24aa1d614172f76c698dd1f021cfa78f44b8db78\": rpc error: code = NotFound desc = could not find container \"5575c0b20a085faef27efaac24aa1d614172f76c698dd1f021cfa78f44b8db78\": container with ID starting with 5575c0b20a085faef27efaac24aa1d614172f76c698dd1f021cfa78f44b8db78 not found: ID does not exist"
Oct 29 22:46:45.122056 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:45.122022    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"]
Oct 29 22:46:45.137751 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:45.137714    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-wsjz8"]
Oct 29 22:46:45.230495 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:45.230464    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5a7c39-6931-4960-932c-910ad8d3c53a" path="/var/lib/kubelet/pods/ca5a7c39-6931-4960-932c-910ad8d3c53a/volumes"
Oct 29 22:46:50.654315 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:50.654279    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"]
Oct 29 22:46:50.655011 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:50.654483    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8" podUID="fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0" containerName="maas-api" containerID="cri-o://3ce2624f1befcf83f9c750a74ddd4a04c4dcfba502fc8d63a96459e8f947badc" gracePeriod=30
Oct 29 22:46:51.620279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:51.620183    4559 generic.go:356] "Generic (PLEG): container finished" podID="fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0" containerID="3ce2624f1befcf83f9c750a74ddd4a04c4dcfba502fc8d63a96459e8f947badc" exitCode=0
Oct 29 22:46:51.622794 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:51.622769    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:46:51.627232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:51.627190    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8" event={"ID":"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0","Type":"ContainerDied","Data":"3ce2624f1befcf83f9c750a74ddd4a04c4dcfba502fc8d63a96459e8f947badc"}
Oct 29 22:46:51.627550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:51.627507    4559 scope.go:122] "RemoveContainer" containerID="3ce2624f1befcf83f9c750a74ddd4a04c4dcfba502fc8d63a96459e8f947badc"
Oct 29 22:46:51.705895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:51.705861    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-openshift-service-ca\") pod \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\" (UID: \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\") "
Oct 29 22:46:51.707030 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:51.706007    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-kube-api-access-z2zb8\" (UniqueName: \"kubernetes.io/projected/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-kube-api-access-z2zb8\") pod \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\" (UID: \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\") "
Oct 29 22:46:51.707030 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:51.706077    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-metrics-certs\") pod \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\" (UID: \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\") "
Oct 29 22:46:51.707030 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:51.706249    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-maas-api-tls\") pod \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\" (UID: \"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0\") "
Oct 29 22:46:51.707030 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:51.706255    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-openshift-service-ca" pod "fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0" (UID: "fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:46:51.708063 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:51.708035    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:51.711050 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:51.711019    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-metrics-certs" pod "fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0" (UID: "fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:46:51.711209 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:51.711148    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-kube-api-access-z2zb8" pod "fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0" (UID: "fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0"). InnerVolumeSpecName "kube-api-access-z2zb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:51.711785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:51.711756    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-maas-api-tls" pod "fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0" (UID: "fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:46:51.809392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:51.809357    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2zb8\" (UniqueName: \"kubernetes.io/projected/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-kube-api-access-z2zb8\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:51.809392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:51.809390    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:51.809392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:51.809399    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:46:52.629615 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:52.629579    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8" event={"ID":"fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0","Type":"ContainerDied","Data":"ac9629d14fc697744213de61f74268cfbdcaebe46cf92df8648a4dbb1ac02c4f"}
Oct 29 22:46:52.629804 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:52.629597    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"
Oct 29 22:46:52.700268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:52.700234    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"]
Oct 29 22:46:52.714955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:52.714917    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1750b-5bddff9dff-4wbt8"]
Oct 29 22:46:53.230863 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:53.230828    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0" path="/var/lib/kubelet/pods/fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0/volumes"
Oct 29 22:46:58.653525 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:58.653481    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"]
Oct 29 22:46:58.654089 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:46:58.653749    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt" podUID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" containerName="istio-proxy" containerID="cri-o://2e680dc60deea7f8d4bcd762190f29259215e753b51099c2451b96c37d1aadf0" gracePeriod=30
Oct 29 22:47:03.787735 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:03.787692    4559 generic.go:356] "Generic (PLEG): container finished" podID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" containerID="2e680dc60deea7f8d4bcd762190f29259215e753b51099c2451b96c37d1aadf0" exitCode=0
Oct 29 22:47:03.788336 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:03.787732    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt" event={"ID":"71bc892b-1fac-4c55-a4c4-493bf662a9ae","Type":"ContainerDied","Data":"2e680dc60deea7f8d4bcd762190f29259215e753b51099c2451b96c37d1aadf0"}
Oct 29 22:47:03.788336 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:03.787863    4559 scope.go:122] "RemoveContainer" containerID="92c0f28d138be4dfb0bfb677c31b3358b72fc95c64844dfb794afc6fa804130a"
Oct 29 22:47:03.975062 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:03.975029    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:47:04.127798 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.127752    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-envoy\") pod \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") "
Oct 29 22:47:04.128037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.127868    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-podinfo\") pod \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") "
Oct 29 22:47:04.128037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.127912    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71bc892b-1fac-4c55-a4c4-493bf662a9ae-kube-api-access-ctfkh\" (UniqueName: \"kubernetes.io/projected/71bc892b-1fac-4c55-a4c4-493bf662a9ae-kube-api-access-ctfkh\") pod \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") "
Oct 29 22:47:04.128037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.127947    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istiod-ca-cert\") pod \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") "
Oct 29 22:47:04.128230 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.128125    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-workload-certs\") pod \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") "
Oct 29 22:47:04.128230 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.128165    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-data\") pod \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") "
Oct 29 22:47:04.128341 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.128260    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-workload-socket\") pod \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") "
Oct 29 22:47:04.128341 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.128299    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-token\" (UniqueName: \"kubernetes.io/projected/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-token\") pod \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") "
Oct 29 22:47:04.128447 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.128378    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istiod-ca-cert" pod "71bc892b-1fac-4c55-a4c4-493bf662a9ae" (UID: "71bc892b-1fac-4c55-a4c4-493bf662a9ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:47:04.128447 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.128392    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-credential-socket\") pod \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\" (UID: \"71bc892b-1fac-4c55-a4c4-493bf662a9ae\") "
Oct 29 22:47:04.128447 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.128375    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-workload-certs" pod "71bc892b-1fac-4c55-a4c4-493bf662a9ae" (UID: "71bc892b-1fac-4c55-a4c4-493bf662a9ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:04.128592 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.128497    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-data" pod "71bc892b-1fac-4c55-a4c4-493bf662a9ae" (UID: "71bc892b-1fac-4c55-a4c4-493bf662a9ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:04.128592 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.128575    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-workload-socket" pod "71bc892b-1fac-4c55-a4c4-493bf662a9ae" (UID: "71bc892b-1fac-4c55-a4c4-493bf662a9ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:04.128865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.128838    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-credential-socket" pod "71bc892b-1fac-4c55-a4c4-493bf662a9ae" (UID: "71bc892b-1fac-4c55-a4c4-493bf662a9ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:04.129551 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.129527    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:47:04.129606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.129559    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:47:04.129606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.129574    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:47:04.129606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.129589    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:47:04.129606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.129603    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:47:04.130660 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.130636    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-token" pod "71bc892b-1fac-4c55-a4c4-493bf662a9ae" (UID: "71bc892b-1fac-4c55-a4c4-493bf662a9ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:04.130760 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.130698    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-envoy" pod "71bc892b-1fac-4c55-a4c4-493bf662a9ae" (UID: "71bc892b-1fac-4c55-a4c4-493bf662a9ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:04.130932 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.130912    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71bc892b-1fac-4c55-a4c4-493bf662a9ae-kube-api-access-ctfkh" pod "71bc892b-1fac-4c55-a4c4-493bf662a9ae" (UID: "71bc892b-1fac-4c55-a4c4-493bf662a9ae"). InnerVolumeSpecName "kube-api-access-ctfkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:04.130932 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.130911    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-podinfo" pod "71bc892b-1fac-4c55-a4c4-493bf662a9ae" (UID: "71bc892b-1fac-4c55-a4c4-493bf662a9ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:47:04.231648 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.231550    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:47:04.231648 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.231584    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctfkh\" (UniqueName: \"kubernetes.io/projected/71bc892b-1fac-4c55-a4c4-493bf662a9ae-kube-api-access-ctfkh\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:47:04.231648 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.231595    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:47:04.231648 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.231604    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71bc892b-1fac-4c55-a4c4-493bf662a9ae-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:47:04.423841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.423794    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:47:04.797804 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.797763    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"
Oct 29 22:47:04.798348 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.797850    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt" event={"ID":"71bc892b-1fac-4c55-a4c4-493bf662a9ae","Type":"ContainerDied","Data":"dc3fc7959fb04792ae4bdb35327075ddc84659da79f5af376c71af5cc1843899"}
Oct 29 22:47:04.798348 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.797915    4559 scope.go:122] "RemoveContainer" containerID="2e680dc60deea7f8d4bcd762190f29259215e753b51099c2451b96c37d1aadf0"
Oct 29 22:47:04.861554 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.861512    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"]
Oct 29 22:47:04.891628 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:04.891586    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c1750b-openshift-default-85d9759655-956xt"]
Oct 29 22:47:05.230954 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:05.230919    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" path="/var/lib/kubelet/pods/71bc892b-1fac-4c55-a4c4-493bf662a9ae/volumes"
Oct 29 22:47:13.699961 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:47:13.699915    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:47:22.867389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.867350    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"]
Oct 29 22:47:22.868227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.868209    4559 cpu_manager.go:414] "containerMap: removing container" podUID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" containerName="istio-proxy"
Oct 29 22:47:22.868227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.868228    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" containerName="istio-proxy"
Oct 29 22:47:22.868312 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.868244    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ca5a7c39-6931-4960-932c-910ad8d3c53a" containerName="istio-proxy"
Oct 29 22:47:22.868312 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.868249    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5a7c39-6931-4960-932c-910ad8d3c53a" containerName="istio-proxy"
Oct 29 22:47:22.868312 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.868264    4559 cpu_manager.go:414] "containerMap: removing container" podUID="fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0" containerName="maas-api"
Oct 29 22:47:22.868312 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.868269    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0" containerName="maas-api"
Oct 29 22:47:22.868457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.868380    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" containerName="istio-proxy"
Oct 29 22:47:22.868457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.868395    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" containerName="istio-proxy"
Oct 29 22:47:22.868457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.868402    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5a7c39-6931-4960-932c-910ad8d3c53a" containerName="istio-proxy"
Oct 29 22:47:22.868457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.868410    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb7e1e83-bb79-4e63-9c0d-4c9112deb2e0" containerName="maas-api"
Oct 29 22:47:22.872554 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.872532    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:22.907787 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.906058    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a5af18-openshift-default-dockercfg-jxpqf\""
Oct 29 22:47:22.908755 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.908722    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"]
Oct 29 22:47:22.978453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.978413    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istiod-ca-cert\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:22.979177 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.979144    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-workload-socket\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:22.979426 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.979405    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fxlg\" (UniqueName: \"kubernetes.io/projected/8c044857-200a-41c6-bbd3-e3f35bf44dc3-kube-api-access-7fxlg\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:22.979589 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.979572    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-workload-certs\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:22.979749 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.979732    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-podinfo\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:22.979944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.979926    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-envoy\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:22.981390 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.980197    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-data\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:22.981632 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.981602    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-token\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:22.981883 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:22.981862    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-credential-socket\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.083721 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.083675    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-workload-socket\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.083886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.083753    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fxlg\" (UniqueName: \"kubernetes.io/projected/8c044857-200a-41c6-bbd3-e3f35bf44dc3-kube-api-access-7fxlg\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.083886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.083782    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-workload-certs\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.084050 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.083898    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-podinfo\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.084146 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.084123    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-workload-socket\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.084213 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.084153    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-workload-certs\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.084272 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.084207    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-envoy\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.084403 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.084382    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-data\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.084484 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.084445    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-token\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.084560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.084546    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-credential-socket\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.084721 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.084699    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-data\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.084794 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.084692    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istiod-ca-cert\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.084938 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.084914    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-credential-socket\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.085445 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.085424    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istiod-ca-cert\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.086745 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.086725    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-envoy\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.087054 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.087032    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-podinfo\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.103578 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.103545    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-token\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.103717 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.103682    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fxlg\" (UniqueName: \"kubernetes.io/projected/8c044857-200a-41c6-bbd3-e3f35bf44dc3-kube-api-access-7fxlg\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.191196 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.191158    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:23.531848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.531795    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"]
Oct 29 22:47:23.532510 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:47:23.532482    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c044857_200a_41c6_bbd3_e3f35bf44dc3.slice/crio-c890212a55197763e1ec4960284dcd7437762cf7770b150fdd805c7fc9e92447 WatchSource:0}: Error finding container c890212a55197763e1ec4960284dcd7437762cf7770b150fdd805c7fc9e92447: Status 404 returned error can't find the container with id c890212a55197763e1ec4960284dcd7437762cf7770b150fdd805c7fc9e92447
Oct 29 22:47:23.535976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.535942    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:47:23.536078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.536013    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:47:23.536078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:23.536049    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:47:24.011860 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:24.011783    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9" event={"ID":"8c044857-200a-41c6-bbd3-e3f35bf44dc3","Type":"ContainerStarted","Data":"2b50627bd60e44daaf7580b890008205a4f0f3c669cd466b5efaebea52f3d582"}
Oct 29 22:47:24.012293 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:24.011887    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9" event={"ID":"8c044857-200a-41c6-bbd3-e3f35bf44dc3","Type":"ContainerStarted","Data":"c890212a55197763e1ec4960284dcd7437762cf7770b150fdd805c7fc9e92447"}
Oct 29 22:47:24.055720 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:24.055663    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9" podStartSLOduration=2.055642352 podStartE2EDuration="2.055642352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:24.054663914 +0000 UTC m=+1982.966107469" watchObservedRunningTime="2026-10-29 22:47:24.055642352 +0000 UTC m=+1982.967085906"
Oct 29 22:47:24.191463 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:24.191424    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:24.193379 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:24.193355    4559 patch_prober.go:29] interesting pod/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 29 22:47:24.193527 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:24.193409    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9" podUID="8c044857-200a-41c6-bbd3-e3f35bf44dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 29 22:47:25.191798 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:25.191751    4559 patch_prober.go:29] interesting pod/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 29 22:47:25.192223 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:25.191855    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9" podUID="8c044857-200a-41c6-bbd3-e3f35bf44dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 29 22:47:26.192387 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:26.192348    4559 patch_prober.go:29] interesting pod/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 29 22:47:26.192928 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:26.192414    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9" podUID="8c044857-200a-41c6-bbd3-e3f35bf44dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 29 22:47:27.192224 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:27.192179    4559 patch_prober.go:29] interesting pod/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 29 22:47:27.192450 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:27.192245    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9" podUID="8c044857-200a-41c6-bbd3-e3f35bf44dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 29 22:47:28.192422 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:28.192382    4559 patch_prober.go:29] interesting pod/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 29 22:47:28.192640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:28.192461    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9" podUID="8c044857-200a-41c6-bbd3-e3f35bf44dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 29 22:47:29.191710 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:29.191669    4559 patch_prober.go:29] interesting pod/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 29 22:47:29.191919 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:29.191727    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9" podUID="8c044857-200a-41c6-bbd3-e3f35bf44dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 29 22:47:30.191802 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:30.191767    4559 patch_prober.go:29] interesting pod/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 29 22:47:30.192226 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:30.191852    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9" podUID="8c044857-200a-41c6-bbd3-e3f35bf44dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 29 22:47:31.196475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:31.196444    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:31.196966 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:31.196943    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:31.197536 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:31.197514    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:47:40.232533 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.232493    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"]
Oct 29 22:47:40.233853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.233792    4559 cpu_manager.go:414] "containerMap: removing container" podUID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" containerName="istio-proxy"
Oct 29 22:47:40.233853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.233836    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71bc892b-1fac-4c55-a4c4-493bf662a9ae" containerName="istio-proxy"
Oct 29 22:47:40.238828 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.238787    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.261162 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.261131    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-d68c2c-openshift-default-dockercfg-28wt6\""
Oct 29 22:47:40.278878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.278836    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-workload-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.279156 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.279131    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-envoy\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.279303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.279264    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-credential-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.279303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.279284    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-workload-certs\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.279425 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.279330    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-podinfo\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.279425 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.279349    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istiod-ca-cert\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.279425 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.279398    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-token\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.279672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.279553    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-data\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.279672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.279578    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxqnd\" (UniqueName: \"kubernetes.io/projected/46bbebaf-946c-4aeb-87f3-eac87926c6ab-kube-api-access-fxqnd\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.284099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.284069    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"]
Oct 29 22:47:40.381217 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.381181    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-podinfo\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.381498 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.381474    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istiod-ca-cert\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.381604 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.381556    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-token\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.381740 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.381725    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-data\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.381889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.381761    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxqnd\" (UniqueName: \"kubernetes.io/projected/46bbebaf-946c-4aeb-87f3-eac87926c6ab-kube-api-access-fxqnd\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.381889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.381846    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-workload-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.382022 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.381930    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-envoy\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.382096 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.382080    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-credential-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.382158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.382115    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-workload-certs\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.382257 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.382231    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istiod-ca-cert\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.382506 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.382480    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-workload-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.382584 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.382484    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-workload-certs\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.382797 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.382772    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-data\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.383321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.383290    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-credential-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.384508 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.384484    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-podinfo\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.384859 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.384796    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-envoy\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.402639 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.402598    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-token\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.403395 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.403375    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxqnd\" (UniqueName: \"kubernetes.io/projected/46bbebaf-946c-4aeb-87f3-eac87926c6ab-kube-api-access-fxqnd\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:40.570707 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:40.568155    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:41.288045 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:41.287998    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:47:41.288663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:41.288579    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:47:41.290792 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:41.290763    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:47:41.304723 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:41.303741    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"]
Oct 29 22:47:41.320055 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:41.320010    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq" event={"ID":"46bbebaf-946c-4aeb-87f3-eac87926c6ab","Type":"ContainerStarted","Data":"dd261ecbaa4632fe6ad70103e51f5b14b0a2a3165799f70f99c79a4552e6c2d6"}
Oct 29 22:47:42.330579 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:42.330544    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq" event={"ID":"46bbebaf-946c-4aeb-87f3-eac87926c6ab","Type":"ContainerStarted","Data":"d50058d742aa421d63bc3046bd9424bd60b7726d34a618a5487cd759002c6719"}
Oct 29 22:47:42.388439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:42.388370    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq" podStartSLOduration=2.388356261 podStartE2EDuration="2.388356261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:42.38443928 +0000 UTC m=+2001.295882835" watchObservedRunningTime="2026-10-29 22:47:42.388356261 +0000 UTC m=+2001.299799814"
Oct 29 22:47:42.578840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:42.571987    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:42.635887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:42.634993    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:43.339097 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:43.339064    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:43.340076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:43.340053    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:47:43.697396 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:47:43.697345    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:47:43.898773 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:43.898737    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"]
Oct 29 22:47:43.899054 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:43.899026    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj" podUID="38c2e753-0737-4b90-a553-428308a0d2ab" containerName="maas-api" containerID="cri-o://969b58ad05620973e6dfbbb71e363e5dc9dd1a69c1e865b0dc79ae7b133dee16" gracePeriod=30
Oct 29 22:47:44.650843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:44.644918    4559 generic.go:356] "Generic (PLEG): container finished" podID="38c2e753-0737-4b90-a553-428308a0d2ab" containerID="969b58ad05620973e6dfbbb71e363e5dc9dd1a69c1e865b0dc79ae7b133dee16" exitCode=0
Oct 29 22:47:44.656836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:44.654543    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj" event={"ID":"38c2e753-0737-4b90-a553-428308a0d2ab","Type":"ContainerDied","Data":"969b58ad05620973e6dfbbb71e363e5dc9dd1a69c1e865b0dc79ae7b133dee16"}
Oct 29 22:47:44.695840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:44.690192    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:47:44.770242 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:44.770183    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c2e753-0737-4b90-a553-428308a0d2ab-openshift-service-ca" pod "38c2e753-0737-4b90-a553-428308a0d2ab" (UID: "38c2e753-0737-4b90-a553-428308a0d2ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:47:44.770539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:44.770519    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38c2e753-0737-4b90-a553-428308a0d2ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c2e753-0737-4b90-a553-428308a0d2ab-openshift-service-ca\") pod \"38c2e753-0737-4b90-a553-428308a0d2ab\" (UID: \"38c2e753-0737-4b90-a553-428308a0d2ab\") "
Oct 29 22:47:44.770834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:44.770797    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38c2e753-0737-4b90-a553-428308a0d2ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c2e753-0737-4b90-a553-428308a0d2ab-metrics-certs\") pod \"38c2e753-0737-4b90-a553-428308a0d2ab\" (UID: \"38c2e753-0737-4b90-a553-428308a0d2ab\") "
Oct 29 22:47:44.771112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:44.771096    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38c2e753-0737-4b90-a553-428308a0d2ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c2e753-0737-4b90-a553-428308a0d2ab-maas-api-tls\") pod \"38c2e753-0737-4b90-a553-428308a0d2ab\" (UID: \"38c2e753-0737-4b90-a553-428308a0d2ab\") "
Oct 29 22:47:44.771318 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:44.771303    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c2e753-0737-4b90-a553-428308a0d2ab-kube-api-access-2794j\" (UniqueName: \"kubernetes.io/projected/38c2e753-0737-4b90-a553-428308a0d2ab-kube-api-access-2794j\") pod \"38c2e753-0737-4b90-a553-428308a0d2ab\" (UID: \"38c2e753-0737-4b90-a553-428308a0d2ab\") "
Oct 29 22:47:44.843360 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:44.843037    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c2e753-0737-4b90-a553-428308a0d2ab-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:47:44.843554 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:44.843498    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c2e753-0737-4b90-a553-428308a0d2ab-kube-api-access-2794j" pod "38c2e753-0737-4b90-a553-428308a0d2ab" (UID: "38c2e753-0737-4b90-a553-428308a0d2ab"). InnerVolumeSpecName "kube-api-access-2794j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:44.843694 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:44.843675    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38c2e753-0737-4b90-a553-428308a0d2ab-maas-api-tls" pod "38c2e753-0737-4b90-a553-428308a0d2ab" (UID: "38c2e753-0737-4b90-a553-428308a0d2ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:47:44.846451 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:44.846411    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38c2e753-0737-4b90-a553-428308a0d2ab-metrics-certs" pod "38c2e753-0737-4b90-a553-428308a0d2ab" (UID: "38c2e753-0737-4b90-a553-428308a0d2ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:47:44.967846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:44.958167    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2794j\" (UniqueName: \"kubernetes.io/projected/38c2e753-0737-4b90-a553-428308a0d2ab-kube-api-access-2794j\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:47:44.967846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:44.958217    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c2e753-0737-4b90-a553-428308a0d2ab-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:47:44.967846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:44.958234    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c2e753-0737-4b90-a553-428308a0d2ab-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:47:45.654636 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:45.654605    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"
Oct 29 22:47:45.654636 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:45.654620    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj" event={"ID":"38c2e753-0737-4b90-a553-428308a0d2ab","Type":"ContainerDied","Data":"c48d1417499ecac8ef250a54513fd12425dd1f9931e2f6d3f6f7ed7c2eb8fae0"}
Oct 29 22:47:45.655255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:45.654686    4559 scope.go:122] "RemoveContainer" containerID="969b58ad05620973e6dfbbb71e363e5dc9dd1a69c1e865b0dc79ae7b133dee16"
Oct 29 22:47:45.721167 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:45.721132    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"]
Oct 29 22:47:45.733292 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:45.733253    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj"]
Oct 29 22:47:47.025042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.025003    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"]
Oct 29 22:47:47.025836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.025804    4559 cpu_manager.go:414] "containerMap: removing container" podUID="38c2e753-0737-4b90-a553-428308a0d2ab" containerName="maas-api"
Oct 29 22:47:47.025836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.025836    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c2e753-0737-4b90-a553-428308a0d2ab" containerName="maas-api"
Oct 29 22:47:47.025997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.025987    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c2e753-0737-4b90-a553-428308a0d2ab" containerName="maas-api"
Oct 29 22:47:47.031152 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.031133    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"
Oct 29 22:47:47.050284 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.050252    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-d68c2c-dockercfg-nd4n2\""
Oct 29 22:47:47.055912 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.055877    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"]
Oct 29 22:47:47.063178 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.063148    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-d68c2c\""
Oct 29 22:47:47.100323 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.100276    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwddh\" (UniqueName: \"kubernetes.io/projected/5db0fef2-9d00-44b3-ab17-70678eddbcda-kube-api-access-zwddh\") pod \"payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r\" (UID: \"5db0fef2-9d00-44b3-ab17-70678eddbcda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"
Oct 29 22:47:47.100657 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.100628    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db0fef2-9d00-44b3-ab17-70678eddbcda-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r\" (UID: \"5db0fef2-9d00-44b3-ab17-70678eddbcda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"
Oct 29 22:47:47.117368 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.117332    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"]
Oct 29 22:47:47.122406 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.122364    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"
Oct 29 22:47:47.243835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.217971    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db0fef2-9d00-44b3-ab17-70678eddbcda-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r\" (UID: \"5db0fef2-9d00-44b3-ab17-70678eddbcda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"
Oct 29 22:47:47.244389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.244353    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8573a8-6d74-4f6d-9c05-11a84c15393b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q\" (UID: \"bc8573a8-6d74-4f6d-9c05-11a84c15393b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"
Oct 29 22:47:47.244651 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.244635    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwddh\" (UniqueName: \"kubernetes.io/projected/5db0fef2-9d00-44b3-ab17-70678eddbcda-kube-api-access-zwddh\") pod \"payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r\" (UID: \"5db0fef2-9d00-44b3-ab17-70678eddbcda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"
Oct 29 22:47:47.244837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.244801    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgmsp\" (UniqueName: \"kubernetes.io/projected/bc8573a8-6d74-4f6d-9c05-11a84c15393b-kube-api-access-qgmsp\") pod \"payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q\" (UID: \"bc8573a8-6d74-4f6d-9c05-11a84c15393b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"
Oct 29 22:47:47.245200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.233959    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db0fef2-9d00-44b3-ab17-70678eddbcda-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r\" (UID: \"5db0fef2-9d00-44b3-ab17-70678eddbcda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"
Oct 29 22:47:47.389715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.381440    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8573a8-6d74-4f6d-9c05-11a84c15393b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q\" (UID: \"bc8573a8-6d74-4f6d-9c05-11a84c15393b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"
Oct 29 22:47:47.389715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.381590    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgmsp\" (UniqueName: \"kubernetes.io/projected/bc8573a8-6d74-4f6d-9c05-11a84c15393b-kube-api-access-qgmsp\") pod \"payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q\" (UID: \"bc8573a8-6d74-4f6d-9c05-11a84c15393b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"
Oct 29 22:47:47.389715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.383489    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8573a8-6d74-4f6d-9c05-11a84c15393b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q\" (UID: \"bc8573a8-6d74-4f6d-9c05-11a84c15393b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"
Oct 29 22:47:47.577888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.555290    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8d985a-85f55b9769-kl9pj" podUID="38c2e753-0737-4b90-a553-428308a0d2ab" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.139:8443/health\": context deadline exceeded"
Oct 29 22:47:47.741837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.680772    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgmsp\" (UniqueName: \"kubernetes.io/projected/bc8573a8-6d74-4f6d-9c05-11a84c15393b-kube-api-access-qgmsp\") pod \"payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q\" (UID: \"bc8573a8-6d74-4f6d-9c05-11a84c15393b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"
Oct 29 22:47:47.787234 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.729958    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38c2e753-0737-4b90-a553-428308a0d2ab" path="/var/lib/kubelet/pods/38c2e753-0737-4b90-a553-428308a0d2ab/volumes"
Oct 29 22:47:47.788080 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.788052    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"]
Oct 29 22:47:47.798024 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.794957    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwddh\" (UniqueName: \"kubernetes.io/projected/5db0fef2-9d00-44b3-ab17-70678eddbcda-kube-api-access-zwddh\") pod \"payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r\" (UID: \"5db0fef2-9d00-44b3-ab17-70678eddbcda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"
Oct 29 22:47:47.943888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.943846    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"
Oct 29 22:47:47.948408 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:47.948384    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"
Oct 29 22:47:48.278113 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:48.278080    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"]
Oct 29 22:47:48.278584 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:47:48.278493    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8573a8_6d74_4f6d_9c05_11a84c15393b.slice/crio-72961d7e15ef4b8465b4122add4a7a2721231a03c7bd061beca4e61b0715a68f WatchSource:0}: Error finding container 72961d7e15ef4b8465b4122add4a7a2721231a03c7bd061beca4e61b0715a68f: Status 404 returned error can't find the container with id 72961d7e15ef4b8465b4122add4a7a2721231a03c7bd061beca4e61b0715a68f
Oct 29 22:47:48.292221 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:48.292187    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"]
Oct 29 22:47:48.297062 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:47:48.297033    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db0fef2_9d00_44b3_ab17_70678eddbcda.slice/crio-36bdae49237718e773805ab3f1f88549e7a35509ec951c95745b3545689a1df0 WatchSource:0}: Error finding container 36bdae49237718e773805ab3f1f88549e7a35509ec951c95745b3545689a1df0: Status 404 returned error can't find the container with id 36bdae49237718e773805ab3f1f88549e7a35509ec951c95745b3545689a1df0
Oct 29 22:47:48.852717 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:48.852674    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r" event={"ID":"5db0fef2-9d00-44b3-ab17-70678eddbcda","Type":"ContainerStarted","Data":"de39b16af5fa8974e6ed99f94fb4a9d60c2febfb0cf010d36d8efa107d3c6abb"}
Oct 29 22:47:48.852934 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:48.852744    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r" event={"ID":"5db0fef2-9d00-44b3-ab17-70678eddbcda","Type":"ContainerStarted","Data":"36bdae49237718e773805ab3f1f88549e7a35509ec951c95745b3545689a1df0"}
Oct 29 22:47:48.854122 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:48.854100    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"
Oct 29 22:47:48.856144 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:48.856117    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q" event={"ID":"bc8573a8-6d74-4f6d-9c05-11a84c15393b","Type":"ContainerStarted","Data":"047bc21fb6737778a4f9f662709dd2163b00903733b0e619fbabdd109635b41f"}
Oct 29 22:47:48.856214 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:48.856172    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q" event={"ID":"bc8573a8-6d74-4f6d-9c05-11a84c15393b","Type":"ContainerStarted","Data":"72961d7e15ef4b8465b4122add4a7a2721231a03c7bd061beca4e61b0715a68f"}
Oct 29 22:47:48.857092 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:48.857074    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"
Oct 29 22:47:48.894511 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:48.894455    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r" podStartSLOduration=2.894442216 podStartE2EDuration="2.894442216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:48.892226955 +0000 UTC m=+2007.803670565" watchObservedRunningTime="2026-10-29 22:47:48.894442216 +0000 UTC m=+2007.805885771"
Oct 29 22:47:48.932646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:48.932588    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q" podStartSLOduration=1.9325713690000001 podStartE2EDuration="1.932571369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:48.931096294 +0000 UTC m=+2007.842539849" watchObservedRunningTime="2026-10-29 22:47:48.932571369 +0000 UTC m=+2007.844014965"
Oct 29 22:47:53.666413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.666365    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"]
Oct 29 22:47:53.674824 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.674512    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.703833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.703779    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"]
Oct 29 22:47:53.709053 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.709006    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-f8ca96-openshift-default-dockercfg-ppssn\""
Oct 29 22:47:53.743318 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.743280    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.743621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.743357    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.743706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.743661    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bde5a64f-e410-46b8-8e6f-c2933360770a-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.743979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.743762    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.743979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.743793    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.743979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.743921    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rggg\" (UniqueName: \"kubernetes.io/projected/bde5a64f-e410-46b8-8e6f-c2933360770a-kube-api-access-5rggg\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.743979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.743952    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.744231 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.744018    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.744231 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.744083    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.864842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.845399    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bde5a64f-e410-46b8-8e6f-c2933360770a-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.864842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.845530    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.864842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.845563    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.864842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.845644    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rggg\" (UniqueName: \"kubernetes.io/projected/bde5a64f-e410-46b8-8e6f-c2933360770a-kube-api-access-5rggg\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.864842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.845672    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.864842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.845742    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.864842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.845829    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.864842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.845906    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.864842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.845967    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.864842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.846575    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.864842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.847315    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bde5a64f-e410-46b8-8e6f-c2933360770a-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.864842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.847615    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.926891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.926095    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:53.964836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:53.953801    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:54.090450 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:54.088629    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rggg\" (UniqueName: \"kubernetes.io/projected/bde5a64f-e410-46b8-8e6f-c2933360770a-kube-api-access-5rggg\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:54.090450 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:54.088883    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:54.130389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:54.130356    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:54.168415 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:54.130982    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:54.308867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:54.308758    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:54.660519 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:47:54.660455    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde5a64f_e410_46b8_8e6f_c2933360770a.slice/crio-1f5b6ac94f56a6eca7b60e3e7137d53b777c574a0766f17aa5a09ef68c808d9c WatchSource:0}: Error finding container 1f5b6ac94f56a6eca7b60e3e7137d53b777c574a0766f17aa5a09ef68c808d9c: Status 404 returned error can't find the container with id 1f5b6ac94f56a6eca7b60e3e7137d53b777c574a0766f17aa5a09ef68c808d9c
Oct 29 22:47:54.660690 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:54.660638    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"]
Oct 29 22:47:54.664006 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:54.663972    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:47:54.664116 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:54.664044    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:47:54.664116 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:54.664082    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:47:55.130669 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:55.130582    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl" event={"ID":"bde5a64f-e410-46b8-8e6f-c2933360770a","Type":"ContainerStarted","Data":"4a2e8cf6f91c20c461350b7aad28501e7d95c5c1389a463b17cbfb527e920529"}
Oct 29 22:47:55.130669 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:55.130634    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl" event={"ID":"bde5a64f-e410-46b8-8e6f-c2933360770a","Type":"ContainerStarted","Data":"1f5b6ac94f56a6eca7b60e3e7137d53b777c574a0766f17aa5a09ef68c808d9c"}
Oct 29 22:47:55.327840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:55.322337    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:55.372836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:55.370185    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:55.421953 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:55.421574    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl" podStartSLOduration=2.421553088 podStartE2EDuration="2.421553088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:55.1983954 +0000 UTC m=+2014.109838955" watchObservedRunningTime="2026-10-29 22:47:55.421553088 +0000 UTC m=+2014.332996642"
Oct 29 22:47:56.140416 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:56.140386    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:47:56.141609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:47:56.141580    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:48:00.877836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:00.874370    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"]
Oct 29 22:48:01.129343 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:01.103491    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"
Oct 29 22:48:01.150978 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:01.142573    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"
Oct 29 22:48:08.064896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:08.064789    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"]
Oct 29 22:48:08.065301 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:08.065021    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" podUID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" containerName="istio-proxy" containerID="cri-o://a54f5da2d7d79612848efedce09b44c55a6546c0882ba30d88b10fcc9bee35c2" gracePeriod=30
Oct 29 22:48:13.483889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.483681    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:48:13.488709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.488660    4559 generic.go:356] "Generic (PLEG): container finished" podID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" containerID="a54f5da2d7d79612848efedce09b44c55a6546c0882ba30d88b10fcc9bee35c2" exitCode=0
Oct 29 22:48:13.489037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.489012    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" event={"ID":"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f","Type":"ContainerDied","Data":"a54f5da2d7d79612848efedce09b44c55a6546c0882ba30d88b10fcc9bee35c2"}
Oct 29 22:48:13.489138 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.489074    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt" event={"ID":"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f","Type":"ContainerDied","Data":"34ddc56898a63d3f77dc2af4ee727ad6cab75831452916d3f709313699b0dbdf"}
Oct 29 22:48:13.489138 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.489107    4559 scope.go:122] "RemoveContainer" containerID="a54f5da2d7d79612848efedce09b44c55a6546c0882ba30d88b10fcc9bee35c2"
Oct 29 22:48:13.489300 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.489284    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"
Oct 29 22:48:13.501946 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.501923    4559 scope.go:122] "RemoveContainer" containerID="a54f5da2d7d79612848efedce09b44c55a6546c0882ba30d88b10fcc9bee35c2"
Oct 29 22:48:13.502273 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:48:13.502249    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54f5da2d7d79612848efedce09b44c55a6546c0882ba30d88b10fcc9bee35c2\": container with ID starting with a54f5da2d7d79612848efedce09b44c55a6546c0882ba30d88b10fcc9bee35c2 not found: ID does not exist" containerID="a54f5da2d7d79612848efedce09b44c55a6546c0882ba30d88b10fcc9bee35c2"
Oct 29 22:48:13.502366 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.502285    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54f5da2d7d79612848efedce09b44c55a6546c0882ba30d88b10fcc9bee35c2"} err="failed to get container status \"a54f5da2d7d79612848efedce09b44c55a6546c0882ba30d88b10fcc9bee35c2\": rpc error: code = NotFound desc = could not find container \"a54f5da2d7d79612848efedce09b44c55a6546c0882ba30d88b10fcc9bee35c2\": container with ID starting with a54f5da2d7d79612848efedce09b44c55a6546c0882ba30d88b10fcc9bee35c2 not found: ID does not exist"
Oct 29 22:48:13.512867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.512840    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-token\" (UniqueName: \"kubernetes.io/projected/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-token\") pod \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") "
Oct 29 22:48:13.512985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.512883    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-data\") pod \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") "
Oct 29 22:48:13.512985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.512952    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-kube-api-access-jvw8f\" (UniqueName: \"kubernetes.io/projected/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-kube-api-access-jvw8f\") pod \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") "
Oct 29 22:48:13.513098 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.513013    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-workload-certs\") pod \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") "
Oct 29 22:48:13.513098 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.513094    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-workload-socket\") pod \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") "
Oct 29 22:48:13.513214 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.513133    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-credential-socket\") pod \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") "
Oct 29 22:48:13.513214 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.513163    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-podinfo\") pod \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") "
Oct 29 22:48:13.513329 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.513221    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-envoy\") pod \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") "
Oct 29 22:48:13.513329 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.513247    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-data" pod "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" (UID: "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:13.513448 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.513365    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istiod-ca-cert\") pod \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\" (UID: \"9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f\") "
Oct 29 22:48:13.513448 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.513377    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-workload-certs" pod "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" (UID: "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:13.513568 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.513474    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-workload-socket" pod "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" (UID: "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:13.513833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.513790    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-credential-socket" pod "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" (UID: "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:13.514010 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.513867    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istiod-ca-cert" pod "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" (UID: "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:13.514419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.514394    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:13.514505 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.514423    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:13.514505 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.514439    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:13.514505 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.514453    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:13.514505 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.514466    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:13.515363 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.515344    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-token" pod "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" (UID: "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:13.515687 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.515663    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-podinfo" pod "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" (UID: "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:48:13.515761 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.515713    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-envoy" pod "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" (UID: "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:13.515935 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.515909    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-kube-api-access-jvw8f" pod "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" (UID: "9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f"). InnerVolumeSpecName "kube-api-access-jvw8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:13.616524 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.616447    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:13.616524 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.616480    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvw8f\" (UniqueName: \"kubernetes.io/projected/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-kube-api-access-jvw8f\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:13.616524 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.616490    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:13.616524 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.616499    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:13.696311 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:48:13.696275    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:48:13.840320 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.840280    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"]
Oct 29 22:48:13.851219 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:13.851185    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-89tnt"]
Oct 29 22:48:15.231248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:15.231215    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" path="/var/lib/kubelet/pods/9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f/volumes"
Oct 29 22:48:16.158005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.157956    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs" podUID="62156571-1397-4073-ab41-ed9b8038f06a" containerName="main" containerID="cri-o://9ec33251e2a8f1c644f4746032db0ce7cade3841efdb1b520a73828be6abbeda" gracePeriod=45
Oct 29 22:48:16.236244 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.236196    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs" podUID="62156571-1397-4073-ab41-ed9b8038f06a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": dial tcp 10.134.0.163:8000: connect: connection refused"
Oct 29 22:48:16.495722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.495696    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:48:16.515544 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.515506    4559 generic.go:356] "Generic (PLEG): container finished" podID="62156571-1397-4073-ab41-ed9b8038f06a" containerID="9ec33251e2a8f1c644f4746032db0ce7cade3841efdb1b520a73828be6abbeda" exitCode=0
Oct 29 22:48:16.515720 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.515559    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"
Oct 29 22:48:16.515720 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.515587    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs" event={"ID":"62156571-1397-4073-ab41-ed9b8038f06a","Type":"ContainerDied","Data":"9ec33251e2a8f1c644f4746032db0ce7cade3841efdb1b520a73828be6abbeda"}
Oct 29 22:48:16.515720 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.515635    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs" event={"ID":"62156571-1397-4073-ab41-ed9b8038f06a","Type":"ContainerDied","Data":"4311b2a9d20f4788ec634d5cc9468c46db36a43c47400f013c0f594d75608639"}
Oct 29 22:48:16.515720 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.515654    4559 scope.go:122] "RemoveContainer" containerID="9ec33251e2a8f1c644f4746032db0ce7cade3841efdb1b520a73828be6abbeda"
Oct 29 22:48:16.526903 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.526871    4559 scope.go:122] "RemoveContainer" containerID="9ec33251e2a8f1c644f4746032db0ce7cade3841efdb1b520a73828be6abbeda"
Oct 29 22:48:16.527289 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:48:16.527270    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec33251e2a8f1c644f4746032db0ce7cade3841efdb1b520a73828be6abbeda\": container with ID starting with 9ec33251e2a8f1c644f4746032db0ce7cade3841efdb1b520a73828be6abbeda not found: ID does not exist" containerID="9ec33251e2a8f1c644f4746032db0ce7cade3841efdb1b520a73828be6abbeda"
Oct 29 22:48:16.527345 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.527299    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec33251e2a8f1c644f4746032db0ce7cade3841efdb1b520a73828be6abbeda"} err="failed to get container status \"9ec33251e2a8f1c644f4746032db0ce7cade3841efdb1b520a73828be6abbeda\": rpc error: code = NotFound desc = could not find container \"9ec33251e2a8f1c644f4746032db0ce7cade3841efdb1b520a73828be6abbeda\": container with ID starting with 9ec33251e2a8f1c644f4746032db0ce7cade3841efdb1b520a73828be6abbeda not found: ID does not exist"
Oct 29 22:48:16.563109 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.563074    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-home\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-home\") pod \"62156571-1397-4073-ab41-ed9b8038f06a\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") "
Oct 29 22:48:16.563321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.563253    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62156571-1397-4073-ab41-ed9b8038f06a-tls-certs\" (UniqueName: \"kubernetes.io/secret/62156571-1397-4073-ab41-ed9b8038f06a-tls-certs\") pod \"62156571-1397-4073-ab41-ed9b8038f06a\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") "
Oct 29 22:48:16.563468 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.563443    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-home" pod "62156571-1397-4073-ab41-ed9b8038f06a" (UID: "62156571-1397-4073-ab41-ed9b8038f06a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:16.563920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.563896    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-dshm\") pod \"62156571-1397-4073-ab41-ed9b8038f06a\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") "
Oct 29 22:48:16.564147 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.564124    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62156571-1397-4073-ab41-ed9b8038f06a-kube-api-access-czdcm\" (UniqueName: \"kubernetes.io/projected/62156571-1397-4073-ab41-ed9b8038f06a-kube-api-access-czdcm\") pod \"62156571-1397-4073-ab41-ed9b8038f06a\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") "
Oct 29 22:48:16.564469 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.564444    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-tmp-dir\") pod \"62156571-1397-4073-ab41-ed9b8038f06a\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") "
Oct 29 22:48:16.564619 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.564601    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-model-cache\") pod \"62156571-1397-4073-ab41-ed9b8038f06a\" (UID: \"62156571-1397-4073-ab41-ed9b8038f06a\") "
Oct 29 22:48:16.564909 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.564885    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-model-cache" pod "62156571-1397-4073-ab41-ed9b8038f06a" (UID: "62156571-1397-4073-ab41-ed9b8038f06a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:16.565006 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.564934    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-tmp-dir" pod "62156571-1397-4073-ab41-ed9b8038f06a" (UID: "62156571-1397-4073-ab41-ed9b8038f06a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:16.565759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.565736    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:16.565907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.565891    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:16.566009 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.565987    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:16.566495 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.566461    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62156571-1397-4073-ab41-ed9b8038f06a-tls-certs" pod "62156571-1397-4073-ab41-ed9b8038f06a" (UID: "62156571-1397-4073-ab41-ed9b8038f06a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:16.569747 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.569721    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-dshm" pod "62156571-1397-4073-ab41-ed9b8038f06a" (UID: "62156571-1397-4073-ab41-ed9b8038f06a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:16.574534 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.574506    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62156571-1397-4073-ab41-ed9b8038f06a-kube-api-access-czdcm" pod "62156571-1397-4073-ab41-ed9b8038f06a" (UID: "62156571-1397-4073-ab41-ed9b8038f06a"). InnerVolumeSpecName "kube-api-access-czdcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:16.705914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.680287    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62156571-1397-4073-ab41-ed9b8038f06a-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:16.705914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.680327    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62156571-1397-4073-ab41-ed9b8038f06a-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:16.705914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.680345    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czdcm\" (UniqueName: \"kubernetes.io/projected/62156571-1397-4073-ab41-ed9b8038f06a-kube-api-access-czdcm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:16.865288 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.865241    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"]
Oct 29 22:48:16.879875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:16.879837    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8147a8/test-model-8147a8-kserve-599675c89c-5xjxs"]
Oct 29 22:48:17.242310 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:17.242262    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62156571-1397-4073-ab41-ed9b8038f06a" path="/var/lib/kubelet/pods/62156571-1397-4073-ab41-ed9b8038f06a/volumes"
Oct 29 22:48:19.395670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:19.395339    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"]
Oct 29 22:48:27.970762 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:27.970724    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:48:29.289834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:29.289482    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"]
Oct 29 22:48:29.824437 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:29.824394    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"]
Oct 29 22:48:34.420084 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.420036    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f" podUID="ea5e69a0-9148-479e-8590-48f3a4848d5b" containerName="main" containerID="cri-o://6cab014def79e249b5990bf1961165c3b830509f15eab74dd10e45f923f94848" gracePeriod=45
Oct 29 22:48:34.542327 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.542286    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f" podUID="ea5e69a0-9148-479e-8590-48f3a4848d5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.164:8000/ready\": dial tcp 10.134.0.164:8000: connect: connection refused"
Oct 29 22:48:34.731550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.731515    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:48:34.752263 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.752229    4559 generic.go:356] "Generic (PLEG): container finished" podID="ea5e69a0-9148-479e-8590-48f3a4848d5b" containerID="6cab014def79e249b5990bf1961165c3b830509f15eab74dd10e45f923f94848" exitCode=0
Oct 29 22:48:34.752461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.752262    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f" event={"ID":"ea5e69a0-9148-479e-8590-48f3a4848d5b","Type":"ContainerDied","Data":"6cab014def79e249b5990bf1961165c3b830509f15eab74dd10e45f923f94848"}
Oct 29 22:48:34.752461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.752293    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"
Oct 29 22:48:34.752461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.752320    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f" event={"ID":"ea5e69a0-9148-479e-8590-48f3a4848d5b","Type":"ContainerDied","Data":"458018d7a2b896c18f03b5a87e42fe8fcedcd051f37892296544eea89a882f5d"}
Oct 29 22:48:34.752461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.752345    4559 scope.go:122] "RemoveContainer" containerID="6cab014def79e249b5990bf1961165c3b830509f15eab74dd10e45f923f94848"
Oct 29 22:48:34.762798 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.762771    4559 scope.go:122] "RemoveContainer" containerID="6cab014def79e249b5990bf1961165c3b830509f15eab74dd10e45f923f94848"
Oct 29 22:48:34.763134 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:48:34.763107    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cab014def79e249b5990bf1961165c3b830509f15eab74dd10e45f923f94848\": container with ID starting with 6cab014def79e249b5990bf1961165c3b830509f15eab74dd10e45f923f94848 not found: ID does not exist" containerID="6cab014def79e249b5990bf1961165c3b830509f15eab74dd10e45f923f94848"
Oct 29 22:48:34.763196 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.763148    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cab014def79e249b5990bf1961165c3b830509f15eab74dd10e45f923f94848"} err="failed to get container status \"6cab014def79e249b5990bf1961165c3b830509f15eab74dd10e45f923f94848\": rpc error: code = NotFound desc = could not find container \"6cab014def79e249b5990bf1961165c3b830509f15eab74dd10e45f923f94848\": container with ID starting with 6cab014def79e249b5990bf1961165c3b830509f15eab74dd10e45f923f94848 not found: ID does not exist"
Oct 29 22:48:34.922437 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.922390    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-tmp-dir\") pod \"ea5e69a0-9148-479e-8590-48f3a4848d5b\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") "
Oct 29 22:48:34.922437 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.922441    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5e69a0-9148-479e-8590-48f3a4848d5b-kube-api-access-69gsk\" (UniqueName: \"kubernetes.io/projected/ea5e69a0-9148-479e-8590-48f3a4848d5b-kube-api-access-69gsk\") pod \"ea5e69a0-9148-479e-8590-48f3a4848d5b\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") "
Oct 29 22:48:34.922702 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.922546    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea5e69a0-9148-479e-8590-48f3a4848d5b-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5e69a0-9148-479e-8590-48f3a4848d5b-tls-certs\") pod \"ea5e69a0-9148-479e-8590-48f3a4848d5b\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") "
Oct 29 22:48:34.922702 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.922660    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-model-cache\") pod \"ea5e69a0-9148-479e-8590-48f3a4848d5b\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") "
Oct 29 22:48:34.922862 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.922698    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-tmp-dir" pod "ea5e69a0-9148-479e-8590-48f3a4848d5b" (UID: "ea5e69a0-9148-479e-8590-48f3a4848d5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:34.922862 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.922720    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-home\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-home\") pod \"ea5e69a0-9148-479e-8590-48f3a4848d5b\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") "
Oct 29 22:48:34.922985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.922946    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-model-cache" pod "ea5e69a0-9148-479e-8590-48f3a4848d5b" (UID: "ea5e69a0-9148-479e-8590-48f3a4848d5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:34.923041 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.922979    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-dshm\") pod \"ea5e69a0-9148-479e-8590-48f3a4848d5b\" (UID: \"ea5e69a0-9148-479e-8590-48f3a4848d5b\") "
Oct 29 22:48:34.923559 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.923531    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-home" pod "ea5e69a0-9148-479e-8590-48f3a4848d5b" (UID: "ea5e69a0-9148-479e-8590-48f3a4848d5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:34.924091 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.924066    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:34.924264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.924239    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:34.924264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.924265    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:34.925433 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.925406    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5e69a0-9148-479e-8590-48f3a4848d5b-tls-certs" pod "ea5e69a0-9148-479e-8590-48f3a4848d5b" (UID: "ea5e69a0-9148-479e-8590-48f3a4848d5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:34.925540 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.925490    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-dshm" pod "ea5e69a0-9148-479e-8590-48f3a4848d5b" (UID: "ea5e69a0-9148-479e-8590-48f3a4848d5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:34.925540 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:34.925532    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5e69a0-9148-479e-8590-48f3a4848d5b-kube-api-access-69gsk" pod "ea5e69a0-9148-479e-8590-48f3a4848d5b" (UID: "ea5e69a0-9148-479e-8590-48f3a4848d5b"). InnerVolumeSpecName "kube-api-access-69gsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:35.026361 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:35.026308    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5e69a0-9148-479e-8590-48f3a4848d5b-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:35.026361 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:35.026363    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69gsk\" (UniqueName: \"kubernetes.io/projected/ea5e69a0-9148-479e-8590-48f3a4848d5b-kube-api-access-69gsk\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:35.026649 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:35.026379    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5e69a0-9148-479e-8590-48f3a4848d5b-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:35.099901 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:35.099865    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"]
Oct 29 22:48:35.104084 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:35.104058    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a5af18/test-model-a5af18-kserve-c978f998c-dr88f"]
Oct 29 22:48:35.231456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:35.231368    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5e69a0-9148-479e-8590-48f3a4848d5b" path="/var/lib/kubelet/pods/ea5e69a0-9148-479e-8590-48f3a4848d5b/volumes"
Oct 29 22:48:43.696492 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:48:43.696455    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:48:44.335017 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.334953    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb" podUID="9779fead-40ad-4cc9-b05e-d2a8d670af27" containerName="main" containerID="cri-o://f239e2e6b1b4c0de5523e931793579956097b135f8198ce44ffdbc26c2b37840" gracePeriod=45
Oct 29 22:48:44.647477 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.647444    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:48:44.756265 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.756217    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-tmp-dir\") pod \"9779fead-40ad-4cc9-b05e-d2a8d670af27\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") "
Oct 29 22:48:44.756776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.756396    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-model-cache\") pod \"9779fead-40ad-4cc9-b05e-d2a8d670af27\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") "
Oct 29 22:48:44.756776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.756451    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-dshm\") pod \"9779fead-40ad-4cc9-b05e-d2a8d670af27\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") "
Oct 29 22:48:44.756776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.756480    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9779fead-40ad-4cc9-b05e-d2a8d670af27-tls-certs\" (UniqueName: \"kubernetes.io/secret/9779fead-40ad-4cc9-b05e-d2a8d670af27-tls-certs\") pod \"9779fead-40ad-4cc9-b05e-d2a8d670af27\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") "
Oct 29 22:48:44.756776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.756519    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-tmp-dir" pod "9779fead-40ad-4cc9-b05e-d2a8d670af27" (UID: "9779fead-40ad-4cc9-b05e-d2a8d670af27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:44.756776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.756588    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-home\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-home\") pod \"9779fead-40ad-4cc9-b05e-d2a8d670af27\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") "
Oct 29 22:48:44.756776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.756638    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-model-cache" pod "9779fead-40ad-4cc9-b05e-d2a8d670af27" (UID: "9779fead-40ad-4cc9-b05e-d2a8d670af27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:44.756776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.756674    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9779fead-40ad-4cc9-b05e-d2a8d670af27-kube-api-access-fmh5q\" (UniqueName: \"kubernetes.io/projected/9779fead-40ad-4cc9-b05e-d2a8d670af27-kube-api-access-fmh5q\") pod \"9779fead-40ad-4cc9-b05e-d2a8d670af27\" (UID: \"9779fead-40ad-4cc9-b05e-d2a8d670af27\") "
Oct 29 22:48:44.757202 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.757127    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-home" pod "9779fead-40ad-4cc9-b05e-d2a8d670af27" (UID: "9779fead-40ad-4cc9-b05e-d2a8d670af27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:44.757845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.757764    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:44.758062 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.758047    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:44.758181 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.758169    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:44.759376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.759332    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-dshm" pod "9779fead-40ad-4cc9-b05e-d2a8d670af27" (UID: "9779fead-40ad-4cc9-b05e-d2a8d670af27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:44.759790 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.759762    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9779fead-40ad-4cc9-b05e-d2a8d670af27-kube-api-access-fmh5q" pod "9779fead-40ad-4cc9-b05e-d2a8d670af27" (UID: "9779fead-40ad-4cc9-b05e-d2a8d670af27"). InnerVolumeSpecName "kube-api-access-fmh5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:44.760658 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.760630    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9779fead-40ad-4cc9-b05e-d2a8d670af27-tls-certs" pod "9779fead-40ad-4cc9-b05e-d2a8d670af27" (UID: "9779fead-40ad-4cc9-b05e-d2a8d670af27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:44.842776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.842738    4559 generic.go:356] "Generic (PLEG): container finished" podID="9779fead-40ad-4cc9-b05e-d2a8d670af27" containerID="f239e2e6b1b4c0de5523e931793579956097b135f8198ce44ffdbc26c2b37840" exitCode=0
Oct 29 22:48:44.842995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.842791    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb" event={"ID":"9779fead-40ad-4cc9-b05e-d2a8d670af27","Type":"ContainerDied","Data":"f239e2e6b1b4c0de5523e931793579956097b135f8198ce44ffdbc26c2b37840"}
Oct 29 22:48:44.842995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.842801    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"
Oct 29 22:48:44.842995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.842852    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb" event={"ID":"9779fead-40ad-4cc9-b05e-d2a8d670af27","Type":"ContainerDied","Data":"5237e380f116b72354e9020d663e0eb3295224cb76e645fc3c120c437d87ddba"}
Oct 29 22:48:44.842995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.842876    4559 scope.go:122] "RemoveContainer" containerID="f239e2e6b1b4c0de5523e931793579956097b135f8198ce44ffdbc26c2b37840"
Oct 29 22:48:44.855866 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.855801    4559 scope.go:122] "RemoveContainer" containerID="f239e2e6b1b4c0de5523e931793579956097b135f8198ce44ffdbc26c2b37840"
Oct 29 22:48:44.856229 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:48:44.856203    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f239e2e6b1b4c0de5523e931793579956097b135f8198ce44ffdbc26c2b37840\": container with ID starting with f239e2e6b1b4c0de5523e931793579956097b135f8198ce44ffdbc26c2b37840 not found: ID does not exist" containerID="f239e2e6b1b4c0de5523e931793579956097b135f8198ce44ffdbc26c2b37840"
Oct 29 22:48:44.856323 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.856239    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f239e2e6b1b4c0de5523e931793579956097b135f8198ce44ffdbc26c2b37840"} err="failed to get container status \"f239e2e6b1b4c0de5523e931793579956097b135f8198ce44ffdbc26c2b37840\": rpc error: code = NotFound desc = could not find container \"f239e2e6b1b4c0de5523e931793579956097b135f8198ce44ffdbc26c2b37840\": container with ID starting with f239e2e6b1b4c0de5523e931793579956097b135f8198ce44ffdbc26c2b37840 not found: ID does not exist"
Oct 29 22:48:44.859432 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.859406    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9779fead-40ad-4cc9-b05e-d2a8d670af27-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:44.859432 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.859433    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9779fead-40ad-4cc9-b05e-d2a8d670af27-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:44.859585 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.859442    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmh5q\" (UniqueName: \"kubernetes.io/projected/9779fead-40ad-4cc9-b05e-d2a8d670af27-kube-api-access-fmh5q\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:44.881554 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.881448    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd" podUID="733bd7c3-3efb-4004-bdd9-10e01133855c" containerName="main" containerID="cri-o://894c7792f40350980fed5a111e9fb92ef2b5a36a413fa65a0a9dc12ee28b34bd" gracePeriod=45
Oct 29 22:48:44.897645 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.897617    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"]
Oct 29 22:48:44.901585 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:44.901549    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5a69c9/test-model-5a69c9-kserve-769b54b4b5-d9wlb"]
Oct 29 22:48:45.165496 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.165464    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:48:45.256836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.253415    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9779fead-40ad-4cc9-b05e-d2a8d670af27" path="/var/lib/kubelet/pods/9779fead-40ad-4cc9-b05e-d2a8d670af27/volumes"
Oct 29 22:48:45.270438 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.270391    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/733bd7c3-3efb-4004-bdd9-10e01133855c-tls-certs\" (UniqueName: \"kubernetes.io/secret/733bd7c3-3efb-4004-bdd9-10e01133855c-tls-certs\") pod \"733bd7c3-3efb-4004-bdd9-10e01133855c\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") "
Oct 29 22:48:45.270625 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.270450    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-tmp-dir\") pod \"733bd7c3-3efb-4004-bdd9-10e01133855c\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") "
Oct 29 22:48:45.270695 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.270675    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-dshm\") pod \"733bd7c3-3efb-4004-bdd9-10e01133855c\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") "
Oct 29 22:48:45.270765 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.270728    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-model-cache\") pod \"733bd7c3-3efb-4004-bdd9-10e01133855c\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") "
Oct 29 22:48:45.270854 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.270838    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-home\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-home\") pod \"733bd7c3-3efb-4004-bdd9-10e01133855c\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") "
Oct 29 22:48:45.270919 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.270888    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733bd7c3-3efb-4004-bdd9-10e01133855c-kube-api-access-8nvqv\" (UniqueName: \"kubernetes.io/projected/733bd7c3-3efb-4004-bdd9-10e01133855c-kube-api-access-8nvqv\") pod \"733bd7c3-3efb-4004-bdd9-10e01133855c\" (UID: \"733bd7c3-3efb-4004-bdd9-10e01133855c\") "
Oct 29 22:48:45.287761 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.287271    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-dshm" pod "733bd7c3-3efb-4004-bdd9-10e01133855c" (UID: "733bd7c3-3efb-4004-bdd9-10e01133855c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:45.289260 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.289223    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-home" pod "733bd7c3-3efb-4004-bdd9-10e01133855c" (UID: "733bd7c3-3efb-4004-bdd9-10e01133855c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:45.290720 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.290490    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-model-cache" pod "733bd7c3-3efb-4004-bdd9-10e01133855c" (UID: "733bd7c3-3efb-4004-bdd9-10e01133855c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:45.290945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.290920    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-tmp-dir" pod "733bd7c3-3efb-4004-bdd9-10e01133855c" (UID: "733bd7c3-3efb-4004-bdd9-10e01133855c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:45.291145 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.290972    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733bd7c3-3efb-4004-bdd9-10e01133855c-kube-api-access-8nvqv" pod "733bd7c3-3efb-4004-bdd9-10e01133855c" (UID: "733bd7c3-3efb-4004-bdd9-10e01133855c"). InnerVolumeSpecName "kube-api-access-8nvqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:45.291260 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.291170    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/733bd7c3-3efb-4004-bdd9-10e01133855c-tls-certs" pod "733bd7c3-3efb-4004-bdd9-10e01133855c" (UID: "733bd7c3-3efb-4004-bdd9-10e01133855c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:45.373199 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.373160    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:45.373199 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.373197    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:45.373455 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.373214    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:45.373455 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.373228    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nvqv\" (UniqueName: \"kubernetes.io/projected/733bd7c3-3efb-4004-bdd9-10e01133855c-kube-api-access-8nvqv\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:45.373455 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.373237    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733bd7c3-3efb-4004-bdd9-10e01133855c-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:45.373455 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.373245    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733bd7c3-3efb-4004-bdd9-10e01133855c-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:45.853791 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.853750    4559 generic.go:356] "Generic (PLEG): container finished" podID="733bd7c3-3efb-4004-bdd9-10e01133855c" containerID="894c7792f40350980fed5a111e9fb92ef2b5a36a413fa65a0a9dc12ee28b34bd" exitCode=0
Oct 29 22:48:45.853791 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.853839    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"
Oct 29 22:48:45.854362 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.853787    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd" event={"ID":"733bd7c3-3efb-4004-bdd9-10e01133855c","Type":"ContainerDied","Data":"894c7792f40350980fed5a111e9fb92ef2b5a36a413fa65a0a9dc12ee28b34bd"}
Oct 29 22:48:45.854362 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.853954    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd" event={"ID":"733bd7c3-3efb-4004-bdd9-10e01133855c","Type":"ContainerDied","Data":"7533038a33fccac5dfe1b3969f590aec88584afb1b35823fa04fa6417223fd3a"}
Oct 29 22:48:45.854362 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.853977    4559 scope.go:122] "RemoveContainer" containerID="894c7792f40350980fed5a111e9fb92ef2b5a36a413fa65a0a9dc12ee28b34bd"
Oct 29 22:48:45.870393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.869958    4559 scope.go:122] "RemoveContainer" containerID="894c7792f40350980fed5a111e9fb92ef2b5a36a413fa65a0a9dc12ee28b34bd"
Oct 29 22:48:45.870713 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:48:45.870541    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"894c7792f40350980fed5a111e9fb92ef2b5a36a413fa65a0a9dc12ee28b34bd\": container with ID starting with 894c7792f40350980fed5a111e9fb92ef2b5a36a413fa65a0a9dc12ee28b34bd not found: ID does not exist" containerID="894c7792f40350980fed5a111e9fb92ef2b5a36a413fa65a0a9dc12ee28b34bd"
Oct 29 22:48:45.870713 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.870624    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894c7792f40350980fed5a111e9fb92ef2b5a36a413fa65a0a9dc12ee28b34bd"} err="failed to get container status \"894c7792f40350980fed5a111e9fb92ef2b5a36a413fa65a0a9dc12ee28b34bd\": rpc error: code = NotFound desc = could not find container \"894c7792f40350980fed5a111e9fb92ef2b5a36a413fa65a0a9dc12ee28b34bd\": container with ID starting with 894c7792f40350980fed5a111e9fb92ef2b5a36a413fa65a0a9dc12ee28b34bd not found: ID does not exist"
Oct 29 22:48:45.905697 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.905654    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"]
Oct 29 22:48:45.924655 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.924617    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"]
Oct 29 22:48:45.930629 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:45.930592    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c7d192/test-model-c7d192-kserve-775b859f67-kgqhd"]
Oct 29 22:48:46.289692 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:46.289641    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"]
Oct 29 22:48:47.235909 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:47.235869    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733bd7c3-3efb-4004-bdd9-10e01133855c" path="/var/lib/kubelet/pods/733bd7c3-3efb-4004-bdd9-10e01133855c/volumes"
Oct 29 22:48:49.907345 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:49.907309    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"]
Oct 29 22:48:49.907865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:49.907588    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd" podUID="ca9a1393-ade2-4ee7-9dd6-a4818740a29c" containerName="maas-api" containerID="cri-o://73557eff956360e8adc519c3ca503c32220041b501a8f24dd2a91c280d004da2" gracePeriod=30
Oct 29 22:48:50.730172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.730146    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:48:50.818375 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.818346    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-kube-api-access-h227s\" (UniqueName: \"kubernetes.io/projected/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-kube-api-access-h227s\") pod \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\" (UID: \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\") "
Oct 29 22:48:50.818573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.818388    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-openshift-service-ca\") pod \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\" (UID: \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\") "
Oct 29 22:48:50.818573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.818508    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-maas-api-tls\") pod \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\" (UID: \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\") "
Oct 29 22:48:50.818573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.818557    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-metrics-certs\") pod \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\" (UID: \"ca9a1393-ade2-4ee7-9dd6-a4818740a29c\") "
Oct 29 22:48:50.818795 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.818761    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-openshift-service-ca" pod "ca9a1393-ade2-4ee7-9dd6-a4818740a29c" (UID: "ca9a1393-ade2-4ee7-9dd6-a4818740a29c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:50.819262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.819220    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:50.820754 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.820729    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-kube-api-access-h227s" pod "ca9a1393-ade2-4ee7-9dd6-a4818740a29c" (UID: "ca9a1393-ade2-4ee7-9dd6-a4818740a29c"). InnerVolumeSpecName "kube-api-access-h227s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:50.820863 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.820774    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-metrics-certs" pod "ca9a1393-ade2-4ee7-9dd6-a4818740a29c" (UID: "ca9a1393-ade2-4ee7-9dd6-a4818740a29c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:50.820914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.820864    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-maas-api-tls" pod "ca9a1393-ade2-4ee7-9dd6-a4818740a29c" (UID: "ca9a1393-ade2-4ee7-9dd6-a4818740a29c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:50.937800 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.937276    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h227s\" (UniqueName: \"kubernetes.io/projected/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-kube-api-access-h227s\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:50.937800 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.937321    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:50.937800 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.937337    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9a1393-ade2-4ee7-9dd6-a4818740a29c-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:48:50.962069 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.961972    4559 generic.go:356] "Generic (PLEG): container finished" podID="ca9a1393-ade2-4ee7-9dd6-a4818740a29c" containerID="73557eff956360e8adc519c3ca503c32220041b501a8f24dd2a91c280d004da2" exitCode=0
Oct 29 22:48:50.962525 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.962498    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd" event={"ID":"ca9a1393-ade2-4ee7-9dd6-a4818740a29c","Type":"ContainerDied","Data":"73557eff956360e8adc519c3ca503c32220041b501a8f24dd2a91c280d004da2"}
Oct 29 22:48:50.962710 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.962687    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd" event={"ID":"ca9a1393-ade2-4ee7-9dd6-a4818740a29c","Type":"ContainerDied","Data":"8d4b87707cfea85f87244efffde49240dda0cf60d9a00b3ea0f6e2a8ea4b8ac0"}
Oct 29 22:48:50.963011 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.962938    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"
Oct 29 22:48:50.965529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.962958    4559 scope.go:122] "RemoveContainer" containerID="73557eff956360e8adc519c3ca503c32220041b501a8f24dd2a91c280d004da2"
Oct 29 22:48:50.982910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.982887    4559 scope.go:122] "RemoveContainer" containerID="73557eff956360e8adc519c3ca503c32220041b501a8f24dd2a91c280d004da2"
Oct 29 22:48:50.983253 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:48:50.983233    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73557eff956360e8adc519c3ca503c32220041b501a8f24dd2a91c280d004da2\": container with ID starting with 73557eff956360e8adc519c3ca503c32220041b501a8f24dd2a91c280d004da2 not found: ID does not exist" containerID="73557eff956360e8adc519c3ca503c32220041b501a8f24dd2a91c280d004da2"
Oct 29 22:48:50.983334 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:50.983261    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73557eff956360e8adc519c3ca503c32220041b501a8f24dd2a91c280d004da2"} err="failed to get container status \"73557eff956360e8adc519c3ca503c32220041b501a8f24dd2a91c280d004da2\": rpc error: code = NotFound desc = could not find container \"73557eff956360e8adc519c3ca503c32220041b501a8f24dd2a91c280d004da2\": container with ID starting with 73557eff956360e8adc519c3ca503c32220041b501a8f24dd2a91c280d004da2 not found: ID does not exist"
Oct 29 22:48:51.025156 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:51.025061    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"]
Oct 29 22:48:51.029497 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:51.029463    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd"]
Oct 29 22:48:51.229803 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:51.229718    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca9a1393-ade2-4ee7-9dd6-a4818740a29c" path="/var/lib/kubelet/pods/ca9a1393-ade2-4ee7-9dd6-a4818740a29c/volumes"
Oct 29 22:48:53.484079 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:53.484036    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fee294-5468577b6-dn5qd" podUID="ca9a1393-ade2-4ee7-9dd6-a4818740a29c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.149:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:48:58.285431 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:58.262736    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:48:58.312499 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:58.312234    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:48:58.333582 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:58.333534    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:48:58.418838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:58.411956    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"]
Oct 29 22:48:58.434039 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:58.433966    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs" podUID="659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" containerName="istio-proxy" containerID="cri-o://38e68ab6cfca49a466d8fe8fa4f314259d91a1e748de9ec37b5f68f5a528f513" gracePeriod=30
Oct 29 22:48:58.614890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:58.593492    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:48:58.662017 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:58.661971    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:48:58.689943 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:58.689897    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:48:58.815883 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:58.787754    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz" podUID="568eefef-fcf7-445b-8dfd-a6b7e79215ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.165:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:48:59.089121 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:48:59.089080    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"]
Oct 29 22:49:00.933776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:00.933736    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz" podUID="568eefef-fcf7-445b-8dfd-a6b7e79215ad" containerName="main" containerID="cri-o://444a483475fa6da5bf47c291077bed93c3555b72b11e6130423ea39f5475d7e2" gracePeriod=45
Oct 29 22:49:01.260260 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.260231    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:49:01.318468 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.318421    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml" podUID="03d29ddc-3695-4fda-ae9d-fb32b252fce0" containerName="main" containerID="cri-o://46e3cffe587c47a1d27d3e716beec6cd49d340a514438c6d975d6a372d63d177" gracePeriod=45
Oct 29 22:49:01.355802 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.355768    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml" podUID="03d29ddc-3695-4fda-ae9d-fb32b252fce0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.178:8000/ready\": dial tcp 10.134.0.178:8000: connect: connection refused"
Oct 29 22:49:01.358272 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.358248    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-dshm\") pod \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") "
Oct 29 22:49:01.358410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.358296    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/568eefef-fcf7-445b-8dfd-a6b7e79215ad-kube-api-access-2bbwp\" (UniqueName: \"kubernetes.io/projected/568eefef-fcf7-445b-8dfd-a6b7e79215ad-kube-api-access-2bbwp\") pod \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") "
Oct 29 22:49:01.358410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.358328    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-home\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-home\") pod \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") "
Oct 29 22:49:01.358539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.358413    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-tmp-dir\") pod \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") "
Oct 29 22:49:01.358539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.358462    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-model-cache\") pod \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") "
Oct 29 22:49:01.358673 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.358643    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/568eefef-fcf7-445b-8dfd-a6b7e79215ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/568eefef-fcf7-445b-8dfd-a6b7e79215ad-tls-certs\") pod \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\" (UID: \"568eefef-fcf7-445b-8dfd-a6b7e79215ad\") "
Oct 29 22:49:01.358739 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.358716    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-tmp-dir" pod "568eefef-fcf7-445b-8dfd-a6b7e79215ad" (UID: "568eefef-fcf7-445b-8dfd-a6b7e79215ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:01.358799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.358736    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-home" pod "568eefef-fcf7-445b-8dfd-a6b7e79215ad" (UID: "568eefef-fcf7-445b-8dfd-a6b7e79215ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:01.358896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.358875    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-model-cache" pod "568eefef-fcf7-445b-8dfd-a6b7e79215ad" (UID: "568eefef-fcf7-445b-8dfd-a6b7e79215ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:01.361159 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.361141    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:01.361159 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.361160    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:01.361344 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.361170    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:01.361541 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.361515    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-dshm" pod "568eefef-fcf7-445b-8dfd-a6b7e79215ad" (UID: "568eefef-fcf7-445b-8dfd-a6b7e79215ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:01.361656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.361589    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568eefef-fcf7-445b-8dfd-a6b7e79215ad-tls-certs" pod "568eefef-fcf7-445b-8dfd-a6b7e79215ad" (UID: "568eefef-fcf7-445b-8dfd-a6b7e79215ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:01.361656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.361606    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568eefef-fcf7-445b-8dfd-a6b7e79215ad-kube-api-access-2bbwp" pod "568eefef-fcf7-445b-8dfd-a6b7e79215ad" (UID: "568eefef-fcf7-445b-8dfd-a6b7e79215ad"). InnerVolumeSpecName "kube-api-access-2bbwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:01.462355 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.462318    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568eefef-fcf7-445b-8dfd-a6b7e79215ad-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:01.462355 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.462353    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568eefef-fcf7-445b-8dfd-a6b7e79215ad-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:01.462558 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.462368    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bbwp\" (UniqueName: \"kubernetes.io/projected/568eefef-fcf7-445b-8dfd-a6b7e79215ad-kube-api-access-2bbwp\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:01.595443 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.595414    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:49:01.665396 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.665357    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-dshm\") pod \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") "
Oct 29 22:49:01.665605 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.665417    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03d29ddc-3695-4fda-ae9d-fb32b252fce0-kube-api-access-t95nk\" (UniqueName: \"kubernetes.io/projected/03d29ddc-3695-4fda-ae9d-fb32b252fce0-kube-api-access-t95nk\") pod \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") "
Oct 29 22:49:01.665685 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.665669    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-tmp-dir\") pod \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") "
Oct 29 22:49:01.665748 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.665703    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-model-cache\") pod \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") "
Oct 29 22:49:01.665748 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.665740    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03d29ddc-3695-4fda-ae9d-fb32b252fce0-tls-certs\" (UniqueName: \"kubernetes.io/secret/03d29ddc-3695-4fda-ae9d-fb32b252fce0-tls-certs\") pod \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") "
Oct 29 22:49:01.665975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.665957    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-home\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-home\") pod \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\" (UID: \"03d29ddc-3695-4fda-ae9d-fb32b252fce0\") "
Oct 29 22:49:01.666057 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.665961    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-tmp-dir" pod "03d29ddc-3695-4fda-ae9d-fb32b252fce0" (UID: "03d29ddc-3695-4fda-ae9d-fb32b252fce0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:01.666128 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.666103    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-model-cache" pod "03d29ddc-3695-4fda-ae9d-fb32b252fce0" (UID: "03d29ddc-3695-4fda-ae9d-fb32b252fce0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:01.666220 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.666201    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-home" pod "03d29ddc-3695-4fda-ae9d-fb32b252fce0" (UID: "03d29ddc-3695-4fda-ae9d-fb32b252fce0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:01.668609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.668579    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:01.668722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.668613    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:01.668722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.668627    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:01.668722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.668626    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-dshm" pod "03d29ddc-3695-4fda-ae9d-fb32b252fce0" (UID: "03d29ddc-3695-4fda-ae9d-fb32b252fce0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:01.669000 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.668974    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d29ddc-3695-4fda-ae9d-fb32b252fce0-kube-api-access-t95nk" pod "03d29ddc-3695-4fda-ae9d-fb32b252fce0" (UID: "03d29ddc-3695-4fda-ae9d-fb32b252fce0"). InnerVolumeSpecName "kube-api-access-t95nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:01.669219 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.669199    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03d29ddc-3695-4fda-ae9d-fb32b252fce0-tls-certs" pod "03d29ddc-3695-4fda-ae9d-fb32b252fce0" (UID: "03d29ddc-3695-4fda-ae9d-fb32b252fce0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:01.730052 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.730016    4559 generic.go:356] "Generic (PLEG): container finished" podID="03d29ddc-3695-4fda-ae9d-fb32b252fce0" containerID="46e3cffe587c47a1d27d3e716beec6cd49d340a514438c6d975d6a372d63d177" exitCode=0
Oct 29 22:49:01.730255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.730062    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"
Oct 29 22:49:01.730255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.730119    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml" event={"ID":"03d29ddc-3695-4fda-ae9d-fb32b252fce0","Type":"ContainerDied","Data":"46e3cffe587c47a1d27d3e716beec6cd49d340a514438c6d975d6a372d63d177"}
Oct 29 22:49:01.730255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.730172    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml" event={"ID":"03d29ddc-3695-4fda-ae9d-fb32b252fce0","Type":"ContainerDied","Data":"c64b058eca8bdd4282e3c16202833d206acd40dce8b228f24d35f6979546ce42"}
Oct 29 22:49:01.730255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.730197    4559 scope.go:122] "RemoveContainer" containerID="46e3cffe587c47a1d27d3e716beec6cd49d340a514438c6d975d6a372d63d177"
Oct 29 22:49:01.731575 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.731556    4559 generic.go:356] "Generic (PLEG): container finished" podID="568eefef-fcf7-445b-8dfd-a6b7e79215ad" containerID="444a483475fa6da5bf47c291077bed93c3555b72b11e6130423ea39f5475d7e2" exitCode=0
Oct 29 22:49:01.731644 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.731624    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz" event={"ID":"568eefef-fcf7-445b-8dfd-a6b7e79215ad","Type":"ContainerDied","Data":"444a483475fa6da5bf47c291077bed93c3555b72b11e6130423ea39f5475d7e2"}
Oct 29 22:49:01.731700 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.731646    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz" event={"ID":"568eefef-fcf7-445b-8dfd-a6b7e79215ad","Type":"ContainerDied","Data":"a6757bd0ae227b388b0dbe24a9479913be5a578b2e03222547cf4469c49af54f"}
Oct 29 22:49:01.731700 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.731697    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"
Oct 29 22:49:01.746352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.746332    4559 scope.go:122] "RemoveContainer" containerID="46e3cffe587c47a1d27d3e716beec6cd49d340a514438c6d975d6a372d63d177"
Oct 29 22:49:01.746730 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:01.746710    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e3cffe587c47a1d27d3e716beec6cd49d340a514438c6d975d6a372d63d177\": container with ID starting with 46e3cffe587c47a1d27d3e716beec6cd49d340a514438c6d975d6a372d63d177 not found: ID does not exist" containerID="46e3cffe587c47a1d27d3e716beec6cd49d340a514438c6d975d6a372d63d177"
Oct 29 22:49:01.746909 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.746733    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e3cffe587c47a1d27d3e716beec6cd49d340a514438c6d975d6a372d63d177"} err="failed to get container status \"46e3cffe587c47a1d27d3e716beec6cd49d340a514438c6d975d6a372d63d177\": rpc error: code = NotFound desc = could not find container \"46e3cffe587c47a1d27d3e716beec6cd49d340a514438c6d975d6a372d63d177\": container with ID starting with 46e3cffe587c47a1d27d3e716beec6cd49d340a514438c6d975d6a372d63d177 not found: ID does not exist"
Oct 29 22:49:01.746909 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.746749    4559 scope.go:122] "RemoveContainer" containerID="444a483475fa6da5bf47c291077bed93c3555b72b11e6130423ea39f5475d7e2"
Oct 29 22:49:01.758281 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.758257    4559 scope.go:122] "RemoveContainer" containerID="444a483475fa6da5bf47c291077bed93c3555b72b11e6130423ea39f5475d7e2"
Oct 29 22:49:01.758613 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:01.758576    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"444a483475fa6da5bf47c291077bed93c3555b72b11e6130423ea39f5475d7e2\": container with ID starting with 444a483475fa6da5bf47c291077bed93c3555b72b11e6130423ea39f5475d7e2 not found: ID does not exist" containerID="444a483475fa6da5bf47c291077bed93c3555b72b11e6130423ea39f5475d7e2"
Oct 29 22:49:01.758698 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.758613    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444a483475fa6da5bf47c291077bed93c3555b72b11e6130423ea39f5475d7e2"} err="failed to get container status \"444a483475fa6da5bf47c291077bed93c3555b72b11e6130423ea39f5475d7e2\": rpc error: code = NotFound desc = could not find container \"444a483475fa6da5bf47c291077bed93c3555b72b11e6130423ea39f5475d7e2\": container with ID starting with 444a483475fa6da5bf47c291077bed93c3555b72b11e6130423ea39f5475d7e2 not found: ID does not exist"
Oct 29 22:49:01.770449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.770417    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03d29ddc-3695-4fda-ae9d-fb32b252fce0-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:01.770449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.770448    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t95nk\" (UniqueName: \"kubernetes.io/projected/03d29ddc-3695-4fda-ae9d-fb32b252fce0-kube-api-access-t95nk\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:01.770642 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.770462    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03d29ddc-3695-4fda-ae9d-fb32b252fce0-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:01.785596 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.785559    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"]
Oct 29 22:49:01.789431 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.789409    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-05417d/test-model-05417d-kserve-86d6967fcc-rtjml"]
Oct 29 22:49:01.829056 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.829021    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"]
Oct 29 22:49:01.832704 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:01.832683    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e619b6/test-model-e619b6-kserve-d94d466d5-97rtz"]
Oct 29 22:49:03.229277 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:03.229239    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03d29ddc-3695-4fda-ae9d-fb32b252fce0" path="/var/lib/kubelet/pods/03d29ddc-3695-4fda-ae9d-fb32b252fce0/volumes"
Oct 29 22:49:03.229756 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:03.229741    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="568eefef-fcf7-445b-8dfd-a6b7e79215ad" path="/var/lib/kubelet/pods/568eefef-fcf7-445b-8dfd-a6b7e79215ad/volumes"
Oct 29 22:49:04.089603 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.089556    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq" podUID="a5f5b015-9b45-407f-ae2b-9d9645b5f09b" containerName="payload-processing" containerID="cri-o://9ea5448f1da7cee51107198c0864e68bae1e77652b15d621f0599d28ac0071b7" gracePeriod=25
Oct 29 22:49:04.122147 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.122120    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:49:04.220282 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.220178    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-token\" (UniqueName: \"kubernetes.io/projected/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-token\") pod \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") "
Oct 29 22:49:04.220282 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.220238    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-workload-socket\") pod \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") "
Oct 29 22:49:04.220525 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.220331    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-envoy\") pod \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") "
Oct 29 22:49:04.220525 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.220392    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-podinfo\") pod \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") "
Oct 29 22:49:04.220525 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.220448    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-kube-api-access-k4wxx\" (UniqueName: \"kubernetes.io/projected/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-kube-api-access-k4wxx\") pod \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") "
Oct 29 22:49:04.220525 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.220512    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istiod-ca-cert\") pod \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") "
Oct 29 22:49:04.220740 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.220550    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-credential-socket\") pod \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") "
Oct 29 22:49:04.220740 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.220717    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-data\") pod \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") "
Oct 29 22:49:04.221926 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.220905    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-workload-certs\") pod \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\" (UID: \"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e\") "
Oct 29 22:49:04.221926 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.220951    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-workload-socket" pod "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" (UID: "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:04.221926 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.221563    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istiod-ca-cert" pod "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" (UID: "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:04.221926 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.221793    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-data" pod "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" (UID: "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:04.221926 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.221841    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:04.221926 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.221864    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:04.222346 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.222025    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-credential-socket" pod "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" (UID: "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:04.222615 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.222594    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-workload-certs" pod "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" (UID: "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:04.223490 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.223462    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-token" pod "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" (UID: "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:04.223989 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.223961    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-podinfo" pod "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" (UID: "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:04.224908 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.224886    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-kube-api-access-k4wxx" pod "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" (UID: "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e"). InnerVolumeSpecName "kube-api-access-k4wxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:04.228076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.228055    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-envoy" pod "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" (UID: "659c8f0f-bcb4-42c8-8c99-8263dfb4b97e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:04.326206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.324689    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:04.326206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.324728    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4wxx\" (UniqueName: \"kubernetes.io/projected/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-kube-api-access-k4wxx\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:04.326206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.324747    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:04.326206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.324763    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:04.326206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.324778    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:04.326206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.324798    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:04.326206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.324836    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:04.791993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.791952    4559 generic.go:356] "Generic (PLEG): container finished" podID="659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" containerID="38e68ab6cfca49a466d8fe8fa4f314259d91a1e748de9ec37b5f68f5a528f513" exitCode=0
Oct 29 22:49:04.792179 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.792011    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"
Oct 29 22:49:04.792179 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.792030    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs" event={"ID":"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e","Type":"ContainerDied","Data":"38e68ab6cfca49a466d8fe8fa4f314259d91a1e748de9ec37b5f68f5a528f513"}
Oct 29 22:49:04.792179 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.792088    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs" event={"ID":"659c8f0f-bcb4-42c8-8c99-8263dfb4b97e","Type":"ContainerDied","Data":"e47d16f49b5c0644432e5c87381aebfd32dd2f8c971447569ac66e30a01e868a"}
Oct 29 22:49:04.792179 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.792110    4559 scope.go:122] "RemoveContainer" containerID="38e68ab6cfca49a466d8fe8fa4f314259d91a1e748de9ec37b5f68f5a528f513"
Oct 29 22:49:04.804860 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.804837    4559 scope.go:122] "RemoveContainer" containerID="38e68ab6cfca49a466d8fe8fa4f314259d91a1e748de9ec37b5f68f5a528f513"
Oct 29 22:49:04.805137 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:04.805118    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e68ab6cfca49a466d8fe8fa4f314259d91a1e748de9ec37b5f68f5a528f513\": container with ID starting with 38e68ab6cfca49a466d8fe8fa4f314259d91a1e748de9ec37b5f68f5a528f513 not found: ID does not exist" containerID="38e68ab6cfca49a466d8fe8fa4f314259d91a1e748de9ec37b5f68f5a528f513"
Oct 29 22:49:04.805193 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.805145    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e68ab6cfca49a466d8fe8fa4f314259d91a1e748de9ec37b5f68f5a528f513"} err="failed to get container status \"38e68ab6cfca49a466d8fe8fa4f314259d91a1e748de9ec37b5f68f5a528f513\": rpc error: code = NotFound desc = could not find container \"38e68ab6cfca49a466d8fe8fa4f314259d91a1e748de9ec37b5f68f5a528f513\": container with ID starting with 38e68ab6cfca49a466d8fe8fa4f314259d91a1e748de9ec37b5f68f5a528f513 not found: ID does not exist"
Oct 29 22:49:04.845913 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.845880    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"]
Oct 29 22:49:04.858522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:04.858490    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-4dbqs"]
Oct 29 22:49:05.322835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:05.316002    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" path="/var/lib/kubelet/pods/659c8f0f-bcb4-42c8-8c99-8263dfb4b97e/volumes"
Oct 29 22:49:06.808932 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:06.808891    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"]
Oct 29 22:49:06.844046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:06.844017    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"]
Oct 29 22:49:10.181624 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.181588    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"]
Oct 29 22:49:10.182386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182364    4559 cpu_manager.go:414] "containerMap: removing container" podUID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" containerName="istio-proxy"
Oct 29 22:49:10.182386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182383    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" containerName="istio-proxy"
Oct 29 22:49:10.182573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182397    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ca9a1393-ade2-4ee7-9dd6-a4818740a29c" containerName="maas-api"
Oct 29 22:49:10.182573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182402    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca9a1393-ade2-4ee7-9dd6-a4818740a29c" containerName="maas-api"
Oct 29 22:49:10.182573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182416    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ea5e69a0-9148-479e-8590-48f3a4848d5b" containerName="main"
Oct 29 22:49:10.182573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182423    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5e69a0-9148-479e-8590-48f3a4848d5b" containerName="main"
Oct 29 22:49:10.182573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182443    4559 cpu_manager.go:414] "containerMap: removing container" podUID="9779fead-40ad-4cc9-b05e-d2a8d670af27" containerName="main"
Oct 29 22:49:10.182573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182448    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9779fead-40ad-4cc9-b05e-d2a8d670af27" containerName="main"
Oct 29 22:49:10.182573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182456    4559 cpu_manager.go:414] "containerMap: removing container" podUID="62156571-1397-4073-ab41-ed9b8038f06a" containerName="main"
Oct 29 22:49:10.182573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182460    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62156571-1397-4073-ab41-ed9b8038f06a" containerName="main"
Oct 29 22:49:10.182573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182468    4559 cpu_manager.go:414] "containerMap: removing container" podUID="03d29ddc-3695-4fda-ae9d-fb32b252fce0" containerName="main"
Oct 29 22:49:10.182573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182473    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d29ddc-3695-4fda-ae9d-fb32b252fce0" containerName="main"
Oct 29 22:49:10.182573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182483    4559 cpu_manager.go:414] "containerMap: removing container" podUID="568eefef-fcf7-445b-8dfd-a6b7e79215ad" containerName="main"
Oct 29 22:49:10.182573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182487    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="568eefef-fcf7-445b-8dfd-a6b7e79215ad" containerName="main"
Oct 29 22:49:10.182573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182494    4559 cpu_manager.go:414] "containerMap: removing container" podUID="659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" containerName="istio-proxy"
Oct 29 22:49:10.182573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182499    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" containerName="istio-proxy"
Oct 29 22:49:10.182573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182527    4559 cpu_manager.go:414] "containerMap: removing container" podUID="733bd7c3-3efb-4004-bdd9-10e01133855c" containerName="main"
Oct 29 22:49:10.182573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182532    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733bd7c3-3efb-4004-bdd9-10e01133855c" containerName="main"
Oct 29 22:49:10.183158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182642    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="568eefef-fcf7-445b-8dfd-a6b7e79215ad" containerName="main"
Oct 29 22:49:10.183158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182656    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="659c8f0f-bcb4-42c8-8c99-8263dfb4b97e" containerName="istio-proxy"
Oct 29 22:49:10.183158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182674    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="733bd7c3-3efb-4004-bdd9-10e01133855c" containerName="main"
Oct 29 22:49:10.183158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182683    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8d9cb1-a201-4c88-b3f2-72e5f2d4ec3f" containerName="istio-proxy"
Oct 29 22:49:10.183158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182691    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca9a1393-ade2-4ee7-9dd6-a4818740a29c" containerName="maas-api"
Oct 29 22:49:10.183158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182702    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5e69a0-9148-479e-8590-48f3a4848d5b" containerName="main"
Oct 29 22:49:10.183158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182717    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="9779fead-40ad-4cc9-b05e-d2a8d670af27" containerName="main"
Oct 29 22:49:10.183158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182726    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="62156571-1397-4073-ab41-ed9b8038f06a" containerName="main"
Oct 29 22:49:10.183158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.182738    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="03d29ddc-3695-4fda-ae9d-fb32b252fce0" containerName="main"
Oct 29 22:49:10.188467 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.188440    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.223207 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.223178    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"]
Oct 29 22:49:10.284565 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.284509    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-credential-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.284786 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.284671    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-workload-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.284786 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.284711    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stfqh\" (UniqueName: \"kubernetes.io/projected/33612869-0569-420d-959a-289d3dcbb1db-kube-api-access-stfqh\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.284998 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.284801    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33612869-0569-420d-959a-289d3dcbb1db-istio-podinfo\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.284998 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.284845    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33612869-0569-420d-959a-289d3dcbb1db-istio-token\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.284998 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.284931    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-istio-data\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.284998 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.284980    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-workload-certs\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.285212 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.285005    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-istio-envoy\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.285212 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.285064    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33612869-0569-420d-959a-289d3dcbb1db-istiod-ca-cert\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.381533 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.381498    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq" podUID="a5f5b015-9b45-407f-ae2b-9d9645b5f09b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:10.386392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.386366    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-workload-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.386500 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.386408    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stfqh\" (UniqueName: \"kubernetes.io/projected/33612869-0569-420d-959a-289d3dcbb1db-kube-api-access-stfqh\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.386500 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.386462    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33612869-0569-420d-959a-289d3dcbb1db-istio-podinfo\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.386500 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.386478    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33612869-0569-420d-959a-289d3dcbb1db-istio-token\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.386625 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.386534    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-istio-data\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.386625 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.386565    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-workload-certs\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.386625 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.386581    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-istio-envoy\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.386775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.386632    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33612869-0569-420d-959a-289d3dcbb1db-istiod-ca-cert\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.386775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.386760    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-credential-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.386901 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.386873    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-workload-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.387088 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.387065    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-workload-certs\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.387177 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.387121    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-credential-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.387177 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.387165    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-istio-data\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.387414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.387395    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33612869-0569-420d-959a-289d3dcbb1db-istiod-ca-cert\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.389612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.389581    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-istio-envoy\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.389718 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.389693    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33612869-0569-420d-959a-289d3dcbb1db-istio-podinfo\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.406394 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.406365    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33612869-0569-420d-959a-289d3dcbb1db-istio-token\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.406560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.406480    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stfqh\" (UniqueName: \"kubernetes.io/projected/33612869-0569-420d-959a-289d3dcbb1db-kube-api-access-stfqh\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.508512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.508419    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:10.862356 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.862324    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"]
Oct 29 22:49:10.864850 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:49:10.864784    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33612869_0569_420d_959a_289d3dcbb1db.slice/crio-6768389651db408d4a9b163ef269c77f60f0994ac2bd2a344f0e16a090966e12 WatchSource:0}: Error finding container 6768389651db408d4a9b163ef269c77f60f0994ac2bd2a344f0e16a090966e12: Status 404 returned error can't find the container with id 6768389651db408d4a9b163ef269c77f60f0994ac2bd2a344f0e16a090966e12
Oct 29 22:49:10.867934 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.867900    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:49:10.868050 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.867984    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:49:10.868050 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.868032    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:49:10.930552 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:10.930514    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp" event={"ID":"33612869-0569-420d-959a-289d3dcbb1db","Type":"ContainerStarted","Data":"6768389651db408d4a9b163ef269c77f60f0994ac2bd2a344f0e16a090966e12"}
Oct 29 22:49:11.812948 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:11.811896    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm" podUID="5a29c9ee-f1a8-460a-a4b7-25625aa72bfd" containerName="payload-processing" containerID="cri-o://6a37d4d91e464cb50096a2f5838e5d8b7d9103645533e9473d3b8dd55f101c59" gracePeriod=25
Oct 29 22:49:11.851102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:11.848885    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp" podUID="068d019b-57d0-4974-9aba-aab57e23b4fb" containerName="payload-pre-processing" containerID="cri-o://4c1d88065c83286b5a0dc5ba5c747bdc3922fca7030a10c6c96edf690d0a42cd" gracePeriod=25
Oct 29 22:49:11.998687 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:11.994726    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp" event={"ID":"33612869-0569-420d-959a-289d3dcbb1db","Type":"ContainerStarted","Data":"1791557a62b92bbae7bc643c3020884e731db344d85773f7d02a1c696424c623"}
Oct 29 22:49:12.095485 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:12.095355    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp" podStartSLOduration=2.095334563 podStartE2EDuration="2.095334563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:12.062299959 +0000 UTC m=+2090.973743514" watchObservedRunningTime="2026-10-29 22:49:12.095334563 +0000 UTC m=+2091.006778122"
Oct 29 22:49:12.411863 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:12.411095    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"
Oct 29 22:49:12.481837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:12.467391    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a29c9ee-f1a8-460a-a4b7-25625aa72bfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a29c9ee-f1a8-460a-a4b7-25625aa72bfd-plugins-config-volume\") pod \"5a29c9ee-f1a8-460a-a4b7-25625aa72bfd\" (UID: \"5a29c9ee-f1a8-460a-a4b7-25625aa72bfd\") "
Oct 29 22:49:12.481837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:12.467620    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a29c9ee-f1a8-460a-a4b7-25625aa72bfd-kube-api-access-s86d2\" (UniqueName: \"kubernetes.io/projected/5a29c9ee-f1a8-460a-a4b7-25625aa72bfd-kube-api-access-s86d2\") pod \"5a29c9ee-f1a8-460a-a4b7-25625aa72bfd\" (UID: \"5a29c9ee-f1a8-460a-a4b7-25625aa72bfd\") "
Oct 29 22:49:12.481837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:12.478287    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a29c9ee-f1a8-460a-a4b7-25625aa72bfd-plugins-config-volume" pod "5a29c9ee-f1a8-460a-a4b7-25625aa72bfd" (UID: "5a29c9ee-f1a8-460a-a4b7-25625aa72bfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:12.510169 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:12.510126    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:12.510975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:12.510941    4559 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:49:12.511138 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:12.511025    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp" podUID="33612869-0569-420d-959a-289d3dcbb1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:49:12.521824 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:12.521748    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a29c9ee-f1a8-460a-a4b7-25625aa72bfd-kube-api-access-s86d2" pod "5a29c9ee-f1a8-460a-a4b7-25625aa72bfd" (UID: "5a29c9ee-f1a8-460a-a4b7-25625aa72bfd"). InnerVolumeSpecName "kube-api-access-s86d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:12.591765 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:12.591727    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s86d2\" (UniqueName: \"kubernetes.io/projected/5a29c9ee-f1a8-460a-a4b7-25625aa72bfd-kube-api-access-s86d2\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:12.592079 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:12.592060    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a29c9ee-f1a8-460a-a4b7-25625aa72bfd-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:13.017491 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:13.017457    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"]
Oct 29 22:49:13.019051 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:13.019017    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9" podUID="8c044857-200a-41c6-bbd3-e3f35bf44dc3" containerName="istio-proxy" containerID="cri-o://2b50627bd60e44daaf7580b890008205a4f0f3c669cd466b5efaebea52f3d582" gracePeriod=30
Oct 29 22:49:13.048904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:13.033250    4559 generic.go:356] "Generic (PLEG): container finished" podID="5a29c9ee-f1a8-460a-a4b7-25625aa72bfd" containerID="6a37d4d91e464cb50096a2f5838e5d8b7d9103645533e9473d3b8dd55f101c59" exitCode=0
Oct 29 22:49:13.048904 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:13.034627    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"
Oct 29 22:49:13.075979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:13.073890    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm" event={"ID":"5a29c9ee-f1a8-460a-a4b7-25625aa72bfd","Type":"ContainerDied","Data":"6a37d4d91e464cb50096a2f5838e5d8b7d9103645533e9473d3b8dd55f101c59"}
Oct 29 22:49:13.075979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:13.073980    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm" event={"ID":"5a29c9ee-f1a8-460a-a4b7-25625aa72bfd","Type":"ContainerDied","Data":"53ec7efb9ab631ed020158adb1c878fa22340e05380205f107f961f11273d3e7"}
Oct 29 22:49:13.075979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:13.074012    4559 scope.go:122] "RemoveContainer" containerID="6a37d4d91e464cb50096a2f5838e5d8b7d9103645533e9473d3b8dd55f101c59"
Oct 29 22:49:13.547452 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:13.540036    4559 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:49:13.547452 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:13.540117    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp" podUID="33612869-0569-420d-959a-289d3dcbb1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:49:13.610153 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:13.610080    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"]
Oct 29 22:49:13.710862 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:13.710824    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c7d192-7f56447bb4-wq8sm"]
Oct 29 22:49:13.731122 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:13.726679    4559 scope.go:122] "RemoveContainer" containerID="6a37d4d91e464cb50096a2f5838e5d8b7d9103645533e9473d3b8dd55f101c59"
Oct 29 22:49:13.731433 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:13.731402    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a37d4d91e464cb50096a2f5838e5d8b7d9103645533e9473d3b8dd55f101c59\": container with ID starting with 6a37d4d91e464cb50096a2f5838e5d8b7d9103645533e9473d3b8dd55f101c59 not found: ID does not exist" containerID="6a37d4d91e464cb50096a2f5838e5d8b7d9103645533e9473d3b8dd55f101c59"
Oct 29 22:49:13.731537 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:13.731447    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a37d4d91e464cb50096a2f5838e5d8b7d9103645533e9473d3b8dd55f101c59"} err="failed to get container status \"6a37d4d91e464cb50096a2f5838e5d8b7d9103645533e9473d3b8dd55f101c59\": rpc error: code = NotFound desc = could not find container \"6a37d4d91e464cb50096a2f5838e5d8b7d9103645533e9473d3b8dd55f101c59\": container with ID starting with 6a37d4d91e464cb50096a2f5838e5d8b7d9103645533e9473d3b8dd55f101c59 not found: ID does not exist"
Oct 29 22:49:13.805034 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:13.804939    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:49:14.509790 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:14.509750    4559 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:49:14.510244 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:14.509841    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp" podUID="33612869-0569-420d-959a-289d3dcbb1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:49:14.667583 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:14.667233    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp" podUID="068d019b-57d0-4974-9aba-aab57e23b4fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:15.229599 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:15.229558    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a29c9ee-f1a8-460a-a4b7-25625aa72bfd" path="/var/lib/kubelet/pods/5a29c9ee-f1a8-460a-a4b7-25625aa72bfd/volumes"
Oct 29 22:49:15.513898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:15.509954    4559 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:49:15.513898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:15.510027    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp" podUID="33612869-0569-420d-959a-289d3dcbb1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:49:16.200045 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:16.200008    4559 patch_prober.go:29] interesting pod/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:49:16.200246 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:16.200073    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9" podUID="8c044857-200a-41c6-bbd3-e3f35bf44dc3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:49:16.497901 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:16.497596    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"]
Oct 29 22:49:16.497901 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:16.497791    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m" podUID="6bc838bd-671f-463d-b98e-0435f56ff70e" containerName="istio-proxy" containerID="cri-o://c8a7ca9b85679ed0ffdda77d56a96a572fc3af922d07a5da6f66a93014d16594" gracePeriod=30
Oct 29 22:49:16.505924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:16.504994    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"]
Oct 29 22:49:16.505924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:16.505237    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx" podUID="702eed95-27cd-42a7-a7fb-f14bcc951474" containerName="istio-proxy" containerID="cri-o://43bc27af0f5395c34baf9822707c89f77e84d0f7787e517d91b93ece94194a51" gracePeriod=30
Oct 29 22:49:16.509731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:16.509338    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"]
Oct 29 22:49:16.509731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:16.509580    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464" podUID="e9e758b8-4105-41d2-a2f6-49e874789dfb" containerName="istio-proxy" containerID="cri-o://92894fb099b8ee7948f9a2bdd37440d1017e638ee9ef3775da8af322cadb14b9" gracePeriod=30
Oct 29 22:49:16.510852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:16.510639    4559 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:49:16.510852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:16.510683    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp" podUID="33612869-0569-420d-959a-289d3dcbb1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:49:17.509255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:17.509214    4559 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:49:17.509832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:17.509304    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp" podUID="33612869-0569-420d-959a-289d3dcbb1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:49:18.624754 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.624725    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:49:18.646588 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.646543    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-data\") pod \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") "
Oct 29 22:49:18.646801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.646670    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-podinfo\") pod \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") "
Oct 29 22:49:18.646801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.646700    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istiod-ca-cert\") pod \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") "
Oct 29 22:49:18.646801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.646747    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-token\" (UniqueName: \"kubernetes.io/projected/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-token\") pod \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") "
Oct 29 22:49:18.646801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.646767    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-workload-socket\") pod \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") "
Oct 29 22:49:18.646801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.646801    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-workload-certs\") pod \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") "
Oct 29 22:49:18.647119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.646861    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-envoy\") pod \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") "
Oct 29 22:49:18.647119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.647041    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-credential-socket\") pod \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") "
Oct 29 22:49:18.647119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.647074    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c044857-200a-41c6-bbd3-e3f35bf44dc3-kube-api-access-7fxlg\" (UniqueName: \"kubernetes.io/projected/8c044857-200a-41c6-bbd3-e3f35bf44dc3-kube-api-access-7fxlg\") pod \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\" (UID: \"8c044857-200a-41c6-bbd3-e3f35bf44dc3\") "
Oct 29 22:49:18.648474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.648424    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-workload-certs" pod "8c044857-200a-41c6-bbd3-e3f35bf44dc3" (UID: "8c044857-200a-41c6-bbd3-e3f35bf44dc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:18.648618 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.648593    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-workload-socket" pod "8c044857-200a-41c6-bbd3-e3f35bf44dc3" (UID: "8c044857-200a-41c6-bbd3-e3f35bf44dc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:18.648696 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.648653    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istiod-ca-cert" pod "8c044857-200a-41c6-bbd3-e3f35bf44dc3" (UID: "8c044857-200a-41c6-bbd3-e3f35bf44dc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:18.648848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.648824    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-data" pod "8c044857-200a-41c6-bbd3-e3f35bf44dc3" (UID: "8c044857-200a-41c6-bbd3-e3f35bf44dc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:18.648914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.648886    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-credential-socket" pod "8c044857-200a-41c6-bbd3-e3f35bf44dc3" (UID: "8c044857-200a-41c6-bbd3-e3f35bf44dc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:18.653030 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.652964    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c044857-200a-41c6-bbd3-e3f35bf44dc3-kube-api-access-7fxlg" pod "8c044857-200a-41c6-bbd3-e3f35bf44dc3" (UID: "8c044857-200a-41c6-bbd3-e3f35bf44dc3"). InnerVolumeSpecName "kube-api-access-7fxlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:18.655044 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.655015    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-token" pod "8c044857-200a-41c6-bbd3-e3f35bf44dc3" (UID: "8c044857-200a-41c6-bbd3-e3f35bf44dc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:18.656262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.656232    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-envoy" pod "8c044857-200a-41c6-bbd3-e3f35bf44dc3" (UID: "8c044857-200a-41c6-bbd3-e3f35bf44dc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:18.657029 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.657006    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-podinfo" pod "8c044857-200a-41c6-bbd3-e3f35bf44dc3" (UID: "8c044857-200a-41c6-bbd3-e3f35bf44dc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:18.723496 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.723459    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:18.725682 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.725644    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:18.726717 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.726696    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:18.749476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.749435    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.750758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.750728    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fxlg\" (UniqueName: \"kubernetes.io/projected/8c044857-200a-41c6-bbd3-e3f35bf44dc3-kube-api-access-7fxlg\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.750994 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.750762    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.750994 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.750795    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.750994 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.750832    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.750994 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.750846    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.750994 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.750858    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.750994 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.750872    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.750994 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:18.750883    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c044857-200a-41c6-bbd3-e3f35bf44dc3-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:19.281827 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:19.281772    4559 generic.go:356] "Generic (PLEG): container finished" podID="8c044857-200a-41c6-bbd3-e3f35bf44dc3" containerID="2b50627bd60e44daaf7580b890008205a4f0f3c669cd466b5efaebea52f3d582" exitCode=0
Oct 29 22:49:19.281981 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:19.281864    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"
Oct 29 22:49:19.281981 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:19.281863    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9" event={"ID":"8c044857-200a-41c6-bbd3-e3f35bf44dc3","Type":"ContainerDied","Data":"2b50627bd60e44daaf7580b890008205a4f0f3c669cd466b5efaebea52f3d582"}
Oct 29 22:49:19.281981 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:19.281925    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9" event={"ID":"8c044857-200a-41c6-bbd3-e3f35bf44dc3","Type":"ContainerDied","Data":"c890212a55197763e1ec4960284dcd7437762cf7770b150fdd805c7fc9e92447"}
Oct 29 22:49:19.281981 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:19.281957    4559 scope.go:122] "RemoveContainer" containerID="2b50627bd60e44daaf7580b890008205a4f0f3c669cd466b5efaebea52f3d582"
Oct 29 22:49:19.296382 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:19.296232    4559 scope.go:122] "RemoveContainer" containerID="2b50627bd60e44daaf7580b890008205a4f0f3c669cd466b5efaebea52f3d582"
Oct 29 22:49:19.296637 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:19.296609    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b50627bd60e44daaf7580b890008205a4f0f3c669cd466b5efaebea52f3d582\": container with ID starting with 2b50627bd60e44daaf7580b890008205a4f0f3c669cd466b5efaebea52f3d582 not found: ID does not exist" containerID="2b50627bd60e44daaf7580b890008205a4f0f3c669cd466b5efaebea52f3d582"
Oct 29 22:49:19.296698 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:19.296651    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b50627bd60e44daaf7580b890008205a4f0f3c669cd466b5efaebea52f3d582"} err="failed to get container status \"2b50627bd60e44daaf7580b890008205a4f0f3c669cd466b5efaebea52f3d582\": rpc error: code = NotFound desc = could not find container \"2b50627bd60e44daaf7580b890008205a4f0f3c669cd466b5efaebea52f3d582\": container with ID starting with 2b50627bd60e44daaf7580b890008205a4f0f3c669cd466b5efaebea52f3d582 not found: ID does not exist"
Oct 29 22:49:19.352129 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:19.352035    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"]
Oct 29 22:49:19.364523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:19.364481    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-7zgh9"]
Oct 29 22:49:20.336764 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:20.336708    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:49:20.339372 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:20.336843    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464" podUID="e9e758b8-4105-41d2-a2f6-49e874789dfb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:49:20.383704 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:20.383665    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq" podUID="a5f5b015-9b45-407f-ae2b-9d9645b5f09b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:20.757248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:20.757208    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:49:20.757424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:20.757276    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx" podUID="702eed95-27cd-42a7-a7fb-f14bcc951474" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:49:21.026531 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:21.026446    4559 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:49:21.026531 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:21.026508    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m" podUID="6bc838bd-671f-463d-b98e-0435f56ff70e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:49:21.229747 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:21.229716    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c044857-200a-41c6-bbd3-e3f35bf44dc3" path="/var/lib/kubelet/pods/8c044857-200a-41c6-bbd3-e3f35bf44dc3/volumes"
Oct 29 22:49:21.316311 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:21.316228    4559 generic.go:356] "Generic (PLEG): container finished" podID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" containerID="d50058d742aa421d63bc3046bd9424bd60b7726d34a618a5487cd759002c6719" exitCode=0
Oct 29 22:49:21.316311 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:21.316269    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq" event={"ID":"46bbebaf-946c-4aeb-87f3-eac87926c6ab","Type":"ContainerDied","Data":"d50058d742aa421d63bc3046bd9424bd60b7726d34a618a5487cd759002c6719"}
Oct 29 22:49:21.316690 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:21.316671    4559 scope.go:122] "RemoveContainer" containerID="d50058d742aa421d63bc3046bd9424bd60b7726d34a618a5487cd759002c6719"
Oct 29 22:49:21.318273 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:21.318233    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:49:21.318327 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:21.318314    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:49:21.318368 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:21.318346    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:49:21.694505 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:21.692612    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:49:21.708488 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:21.708457    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:49:22.128914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.119696    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:49:22.152766 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.152733    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.320976    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-token\" (UniqueName: \"kubernetes.io/projected/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-token\") pod \"702eed95-27cd-42a7-a7fb-f14bcc951474\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.321070    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/702eed95-27cd-42a7-a7fb-f14bcc951474-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702eed95-27cd-42a7-a7fb-f14bcc951474-istiod-ca-cert\") pod \"702eed95-27cd-42a7-a7fb-f14bcc951474\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.321189    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-envoy\") pod \"702eed95-27cd-42a7-a7fb-f14bcc951474\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.321222    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-data\") pod \"e9e758b8-4105-41d2-a2f6-49e874789dfb\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.321265    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-credential-socket\") pod \"e9e758b8-4105-41d2-a2f6-49e874789dfb\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.321295    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e758b8-4105-41d2-a2f6-49e874789dfb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e758b8-4105-41d2-a2f6-49e874789dfb-istiod-ca-cert\") pod \"e9e758b8-4105-41d2-a2f6-49e874789dfb\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.321359    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-podinfo\") pod \"e9e758b8-4105-41d2-a2f6-49e874789dfb\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.321406    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702eed95-27cd-42a7-a7fb-f14bcc951474-kube-api-access-bc98s\" (UniqueName: \"kubernetes.io/projected/702eed95-27cd-42a7-a7fb-f14bcc951474-kube-api-access-bc98s\") pod \"702eed95-27cd-42a7-a7fb-f14bcc951474\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.321504    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-envoy\") pod \"e9e758b8-4105-41d2-a2f6-49e874789dfb\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.321540    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-data\") pod \"702eed95-27cd-42a7-a7fb-f14bcc951474\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.321582    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-workload-certs\") pod \"e9e758b8-4105-41d2-a2f6-49e874789dfb\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.321617    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-workload-socket\") pod \"702eed95-27cd-42a7-a7fb-f14bcc951474\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.321667    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-credential-socket\") pod \"702eed95-27cd-42a7-a7fb-f14bcc951474\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.321759    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-workload-socket\") pod \"e9e758b8-4105-41d2-a2f6-49e874789dfb\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.321858    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-token\" (UniqueName: \"kubernetes.io/projected/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-token\") pod \"e9e758b8-4105-41d2-a2f6-49e874789dfb\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.321889    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-workload-certs\") pod \"702eed95-27cd-42a7-a7fb-f14bcc951474\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.321917    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-podinfo\") pod \"702eed95-27cd-42a7-a7fb-f14bcc951474\" (UID: \"702eed95-27cd-42a7-a7fb-f14bcc951474\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.321955    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e758b8-4105-41d2-a2f6-49e874789dfb-kube-api-access-ltmft\" (UniqueName: \"kubernetes.io/projected/e9e758b8-4105-41d2-a2f6-49e874789dfb-kube-api-access-ltmft\") pod \"e9e758b8-4105-41d2-a2f6-49e874789dfb\" (UID: \"e9e758b8-4105-41d2-a2f6-49e874789dfb\") "
Oct 29 22:49:22.325922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.325421    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-data" pod "e9e758b8-4105-41d2-a2f6-49e874789dfb" (UID: "e9e758b8-4105-41d2-a2f6-49e874789dfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.327097 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.326009    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e758b8-4105-41d2-a2f6-49e874789dfb-istiod-ca-cert" pod "e9e758b8-4105-41d2-a2f6-49e874789dfb" (UID: "e9e758b8-4105-41d2-a2f6-49e874789dfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:22.327097 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.326315    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-credential-socket" pod "e9e758b8-4105-41d2-a2f6-49e874789dfb" (UID: "e9e758b8-4105-41d2-a2f6-49e874789dfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.327097 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.326597    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/702eed95-27cd-42a7-a7fb-f14bcc951474-istiod-ca-cert" pod "702eed95-27cd-42a7-a7fb-f14bcc951474" (UID: "702eed95-27cd-42a7-a7fb-f14bcc951474"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:22.343419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.327131    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-data" pod "702eed95-27cd-42a7-a7fb-f14bcc951474" (UID: "702eed95-27cd-42a7-a7fb-f14bcc951474"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.343419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.327377    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-workload-socket" pod "e9e758b8-4105-41d2-a2f6-49e874789dfb" (UID: "e9e758b8-4105-41d2-a2f6-49e874789dfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.343419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.327693    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-workload-certs" pod "702eed95-27cd-42a7-a7fb-f14bcc951474" (UID: "702eed95-27cd-42a7-a7fb-f14bcc951474"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.343419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.327701    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-workload-certs" pod "e9e758b8-4105-41d2-a2f6-49e874789dfb" (UID: "e9e758b8-4105-41d2-a2f6-49e874789dfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.343419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.327904    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-workload-socket" pod "702eed95-27cd-42a7-a7fb-f14bcc951474" (UID: "702eed95-27cd-42a7-a7fb-f14bcc951474"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.343419 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.328075    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-credential-socket" pod "702eed95-27cd-42a7-a7fb-f14bcc951474" (UID: "702eed95-27cd-42a7-a7fb-f14bcc951474"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.353982 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.351312    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-token" pod "e9e758b8-4105-41d2-a2f6-49e874789dfb" (UID: "e9e758b8-4105-41d2-a2f6-49e874789dfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:22.355436 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.355042    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-podinfo" pod "702eed95-27cd-42a7-a7fb-f14bcc951474" (UID: "702eed95-27cd-42a7-a7fb-f14bcc951474"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:22.355436 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.355134    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702eed95-27cd-42a7-a7fb-f14bcc951474-kube-api-access-bc98s" pod "702eed95-27cd-42a7-a7fb-f14bcc951474" (UID: "702eed95-27cd-42a7-a7fb-f14bcc951474"). InnerVolumeSpecName "kube-api-access-bc98s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:22.357195 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.357154    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-podinfo" pod "e9e758b8-4105-41d2-a2f6-49e874789dfb" (UID: "e9e758b8-4105-41d2-a2f6-49e874789dfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:22.359170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.357420    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-token" pod "702eed95-27cd-42a7-a7fb-f14bcc951474" (UID: "702eed95-27cd-42a7-a7fb-f14bcc951474"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:22.359170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.357534    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-envoy" pod "e9e758b8-4105-41d2-a2f6-49e874789dfb" (UID: "e9e758b8-4105-41d2-a2f6-49e874789dfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.359170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.357641    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e758b8-4105-41d2-a2f6-49e874789dfb-kube-api-access-ltmft" pod "e9e758b8-4105-41d2-a2f6-49e874789dfb" (UID: "e9e758b8-4105-41d2-a2f6-49e874789dfb"). InnerVolumeSpecName "kube-api-access-ltmft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:22.359170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.358014    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq" event={"ID":"46bbebaf-946c-4aeb-87f3-eac87926c6ab","Type":"ContainerStarted","Data":"bde9e4588b334cc02e0e4f057ef4318001071585fdb3e1717622738a21736b29"}
Oct 29 22:49:22.362355 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.361961    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-envoy" pod "702eed95-27cd-42a7-a7fb-f14bcc951474" (UID: "702eed95-27cd-42a7-a7fb-f14bcc951474"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.366461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.365802    4559 generic.go:356] "Generic (PLEG): container finished" podID="6bc838bd-671f-463d-b98e-0435f56ff70e" containerID="c8a7ca9b85679ed0ffdda77d56a96a572fc3af922d07a5da6f66a93014d16594" exitCode=0
Oct 29 22:49:22.366461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.365929    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m" event={"ID":"6bc838bd-671f-463d-b98e-0435f56ff70e","Type":"ContainerDied","Data":"c8a7ca9b85679ed0ffdda77d56a96a572fc3af922d07a5da6f66a93014d16594"}
Oct 29 22:49:22.368393 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.368371    4559 generic.go:356] "Generic (PLEG): container finished" podID="702eed95-27cd-42a7-a7fb-f14bcc951474" containerID="43bc27af0f5395c34baf9822707c89f77e84d0f7787e517d91b93ece94194a51" exitCode=0
Oct 29 22:49:22.368493 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.368478    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx" event={"ID":"702eed95-27cd-42a7-a7fb-f14bcc951474","Type":"ContainerDied","Data":"43bc27af0f5395c34baf9822707c89f77e84d0f7787e517d91b93ece94194a51"}
Oct 29 22:49:22.368543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.368516    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx" event={"ID":"702eed95-27cd-42a7-a7fb-f14bcc951474","Type":"ContainerDied","Data":"a90e18bc5b9f418eed5a1b23e0ebc7629a957d0dd4d79474e26d5ce45a7511c8"}
Oct 29 22:49:22.368543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.368537    4559 scope.go:122] "RemoveContainer" containerID="43bc27af0f5395c34baf9822707c89f77e84d0f7787e517d91b93ece94194a51"
Oct 29 22:49:22.368691 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.368676    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"
Oct 29 22:49:22.372797 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.371382    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:49:22.375775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.375742    4559 generic.go:356] "Generic (PLEG): container finished" podID="e9e758b8-4105-41d2-a2f6-49e874789dfb" containerID="92894fb099b8ee7948f9a2bdd37440d1017e638ee9ef3775da8af322cadb14b9" exitCode=0
Oct 29 22:49:22.375932 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.375859    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464" event={"ID":"e9e758b8-4105-41d2-a2f6-49e874789dfb","Type":"ContainerDied","Data":"92894fb099b8ee7948f9a2bdd37440d1017e638ee9ef3775da8af322cadb14b9"}
Oct 29 22:49:22.375932 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.375908    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464" event={"ID":"e9e758b8-4105-41d2-a2f6-49e874789dfb","Type":"ContainerDied","Data":"f23abf36a131293300984d34f895892a286dd958ddad7a9a044e2eb6b99c1523"}
Oct 29 22:49:22.376063 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.375982    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"
Oct 29 22:49:22.407503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.407477    4559 scope.go:122] "RemoveContainer" containerID="43bc27af0f5395c34baf9822707c89f77e84d0f7787e517d91b93ece94194a51"
Oct 29 22:49:22.408677 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:22.408646    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43bc27af0f5395c34baf9822707c89f77e84d0f7787e517d91b93ece94194a51\": container with ID starting with 43bc27af0f5395c34baf9822707c89f77e84d0f7787e517d91b93ece94194a51 not found: ID does not exist" containerID="43bc27af0f5395c34baf9822707c89f77e84d0f7787e517d91b93ece94194a51"
Oct 29 22:49:22.408843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.408689    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43bc27af0f5395c34baf9822707c89f77e84d0f7787e517d91b93ece94194a51"} err="failed to get container status \"43bc27af0f5395c34baf9822707c89f77e84d0f7787e517d91b93ece94194a51\": rpc error: code = NotFound desc = could not find container \"43bc27af0f5395c34baf9822707c89f77e84d0f7787e517d91b93ece94194a51\": container with ID starting with 43bc27af0f5395c34baf9822707c89f77e84d0f7787e517d91b93ece94194a51 not found: ID does not exist"
Oct 29 22:49:22.408843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.408713    4559 scope.go:122] "RemoveContainer" containerID="92894fb099b8ee7948f9a2bdd37440d1017e638ee9ef3775da8af322cadb14b9"
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.424772    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc838bd-671f-463d-b98e-0435f56ff70e-kube-api-access-278wj\" (UniqueName: \"kubernetes.io/projected/6bc838bd-671f-463d-b98e-0435f56ff70e-kube-api-access-278wj\") pod \"6bc838bd-671f-463d-b98e-0435f56ff70e\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") "
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.424985    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-token\" (UniqueName: \"kubernetes.io/projected/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-token\") pod \"6bc838bd-671f-463d-b98e-0435f56ff70e\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") "
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.425074    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-envoy\") pod \"6bc838bd-671f-463d-b98e-0435f56ff70e\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") "
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.425130    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-credential-socket\") pod \"6bc838bd-671f-463d-b98e-0435f56ff70e\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") "
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.425180    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-workload-socket\") pod \"6bc838bd-671f-463d-b98e-0435f56ff70e\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") "
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.425257    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-workload-certs\") pod \"6bc838bd-671f-463d-b98e-0435f56ff70e\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") "
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.425324    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-data\") pod \"6bc838bd-671f-463d-b98e-0435f56ff70e\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") "
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.425409    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc838bd-671f-463d-b98e-0435f56ff70e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc838bd-671f-463d-b98e-0435f56ff70e-istiod-ca-cert\") pod \"6bc838bd-671f-463d-b98e-0435f56ff70e\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") "
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.425437    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-podinfo\") pod \"6bc838bd-671f-463d-b98e-0435f56ff70e\" (UID: \"6bc838bd-671f-463d-b98e-0435f56ff70e\") "
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426164    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702eed95-27cd-42a7-a7fb-f14bcc951474-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426179    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426190    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426199    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426206    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e758b8-4105-41d2-a2f6-49e874789dfb-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426214    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426222    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc98s\" (UniqueName: \"kubernetes.io/projected/702eed95-27cd-42a7-a7fb-f14bcc951474-kube-api-access-bc98s\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426231    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426239    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426248    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426258    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426265    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426273    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e758b8-4105-41d2-a2f6-49e874789dfb-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426281    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e758b8-4105-41d2-a2f6-49e874789dfb-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426288    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702eed95-27cd-42a7-a7fb-f14bcc951474-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426296    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426303    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltmft\" (UniqueName: \"kubernetes.io/projected/e9e758b8-4105-41d2-a2f6-49e874789dfb-kube-api-access-ltmft\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.426311    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702eed95-27cd-42a7-a7fb-f14bcc951474-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.427118    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-data" pod "6bc838bd-671f-463d-b98e-0435f56ff70e" (UID: "6bc838bd-671f-463d-b98e-0435f56ff70e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.427978    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc838bd-671f-463d-b98e-0435f56ff70e-istiod-ca-cert" pod "6bc838bd-671f-463d-b98e-0435f56ff70e" (UID: "6bc838bd-671f-463d-b98e-0435f56ff70e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.428584    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-credential-socket" pod "6bc838bd-671f-463d-b98e-0435f56ff70e" (UID: "6bc838bd-671f-463d-b98e-0435f56ff70e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.429096    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-workload-socket" pod "6bc838bd-671f-463d-b98e-0435f56ff70e" (UID: "6bc838bd-671f-463d-b98e-0435f56ff70e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.429838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.429145    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-workload-certs" pod "6bc838bd-671f-463d-b98e-0435f56ff70e" (UID: "6bc838bd-671f-463d-b98e-0435f56ff70e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.431672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.431054    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-podinfo" pod "6bc838bd-671f-463d-b98e-0435f56ff70e" (UID: "6bc838bd-671f-463d-b98e-0435f56ff70e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:22.432480 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.432445    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-envoy" pod "6bc838bd-671f-463d-b98e-0435f56ff70e" (UID: "6bc838bd-671f-463d-b98e-0435f56ff70e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.467985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.467900    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-token" pod "6bc838bd-671f-463d-b98e-0435f56ff70e" (UID: "6bc838bd-671f-463d-b98e-0435f56ff70e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:22.469868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.469035    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc838bd-671f-463d-b98e-0435f56ff70e-kube-api-access-278wj" pod "6bc838bd-671f-463d-b98e-0435f56ff70e" (UID: "6bc838bd-671f-463d-b98e-0435f56ff70e"). InnerVolumeSpecName "kube-api-access-278wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:22.472688 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.472651    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"]
Oct 29 22:49:22.473564 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.473539    4559 scope.go:122] "RemoveContainer" containerID="92894fb099b8ee7948f9a2bdd37440d1017e638ee9ef3775da8af322cadb14b9"
Oct 29 22:49:22.473947 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:22.473917    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92894fb099b8ee7948f9a2bdd37440d1017e638ee9ef3775da8af322cadb14b9\": container with ID starting with 92894fb099b8ee7948f9a2bdd37440d1017e638ee9ef3775da8af322cadb14b9 not found: ID does not exist" containerID="92894fb099b8ee7948f9a2bdd37440d1017e638ee9ef3775da8af322cadb14b9"
Oct 29 22:49:22.474018 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.473958    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92894fb099b8ee7948f9a2bdd37440d1017e638ee9ef3775da8af322cadb14b9"} err="failed to get container status \"92894fb099b8ee7948f9a2bdd37440d1017e638ee9ef3775da8af322cadb14b9\": rpc error: code = NotFound desc = could not find container \"92894fb099b8ee7948f9a2bdd37440d1017e638ee9ef3775da8af322cadb14b9\": container with ID starting with 92894fb099b8ee7948f9a2bdd37440d1017e638ee9ef3775da8af322cadb14b9 not found: ID does not exist"
Oct 29 22:49:22.476353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.476331    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-4krkx"]
Oct 29 22:49:22.509399 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.509365    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"]
Oct 29 22:49:22.513010 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.512990    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-kl464"]
Oct 29 22:49:22.528357 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.528330    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.528357 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.528356    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.528529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.528365    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.528529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.528374    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.528529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.528387    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.528529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.528397    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc838bd-671f-463d-b98e-0435f56ff70e-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.528529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.528406    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.528529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.528414    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-278wj\" (UniqueName: \"kubernetes.io/projected/6bc838bd-671f-463d-b98e-0435f56ff70e-kube-api-access-278wj\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.528529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.528422    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc838bd-671f-463d-b98e-0435f56ff70e-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.571223 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.571067    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:49:22.572398 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.572254    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 29 22:49:22.572398 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:22.572305    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq" podUID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 29 22:49:23.109658 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:23.109626    4559 scope.go:122] "RemoveContainer" containerID="c8a7ca9b85679ed0ffdda77d56a96a572fc3af922d07a5da6f66a93014d16594"
Oct 29 22:49:23.272874 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:23.232900    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="702eed95-27cd-42a7-a7fb-f14bcc951474" path="/var/lib/kubelet/pods/702eed95-27cd-42a7-a7fb-f14bcc951474/volumes"
Oct 29 22:49:23.272874 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:23.233646    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e758b8-4105-41d2-a2f6-49e874789dfb" path="/var/lib/kubelet/pods/e9e758b8-4105-41d2-a2f6-49e874789dfb/volumes"
Oct 29 22:49:23.385090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:23.385007    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m" event={"ID":"6bc838bd-671f-463d-b98e-0435f56ff70e","Type":"ContainerDied","Data":"1e85493a94f69622ef354206c6c088e2ec5b865683c8549dbf217d2bec8bad34"}
Oct 29 22:49:23.385246 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:23.385146    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"
Oct 29 22:49:23.494327 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:23.494291    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"]
Oct 29 22:49:23.497758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:23.497733    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-xw24m"]
Oct 29 22:49:23.569316 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:23.569273    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 29 22:49:23.569523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:23.569345    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq" podUID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 29 22:49:24.418928 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:24.418885    4559 generic.go:356] "Generic (PLEG): container finished" podID="a5f5b015-9b45-407f-ae2b-9d9645b5f09b" containerID="9ea5448f1da7cee51107198c0864e68bae1e77652b15d621f0599d28ac0071b7" exitCode=0
Oct 29 22:49:24.419520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:24.419029    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq" event={"ID":"a5f5b015-9b45-407f-ae2b-9d9645b5f09b","Type":"ContainerDied","Data":"9ea5448f1da7cee51107198c0864e68bae1e77652b15d621f0599d28ac0071b7"}
Oct 29 22:49:24.573912 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:24.572229    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 29 22:49:24.573912 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:24.572304    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq" podUID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 29 22:49:24.610258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:24.610225    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"
Oct 29 22:49:24.765914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:24.763195    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f5b015-9b45-407f-ae2b-9d9645b5f09b-kube-api-access-sg5h5\" (UniqueName: \"kubernetes.io/projected/a5f5b015-9b45-407f-ae2b-9d9645b5f09b-kube-api-access-sg5h5\") pod \"a5f5b015-9b45-407f-ae2b-9d9645b5f09b\" (UID: \"a5f5b015-9b45-407f-ae2b-9d9645b5f09b\") "
Oct 29 22:49:24.765914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:24.763532    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f5b015-9b45-407f-ae2b-9d9645b5f09b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f5b015-9b45-407f-ae2b-9d9645b5f09b-plugins-config-volume\") pod \"a5f5b015-9b45-407f-ae2b-9d9645b5f09b\" (UID: \"a5f5b015-9b45-407f-ae2b-9d9645b5f09b\") "
Oct 29 22:49:24.812865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:24.802448    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f5b015-9b45-407f-ae2b-9d9645b5f09b-kube-api-access-sg5h5" pod "a5f5b015-9b45-407f-ae2b-9d9645b5f09b" (UID: "a5f5b015-9b45-407f-ae2b-9d9645b5f09b"). InnerVolumeSpecName "kube-api-access-sg5h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:24.876840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:24.873119    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg5h5\" (UniqueName: \"kubernetes.io/projected/a5f5b015-9b45-407f-ae2b-9d9645b5f09b-kube-api-access-sg5h5\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:24.931608 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:24.929995    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp" podUID="068d019b-57d0-4974-9aba-aab57e23b4fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:24.985252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:24.985190    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f5b015-9b45-407f-ae2b-9d9645b5f09b-plugins-config-volume" pod "a5f5b015-9b45-407f-ae2b-9d9645b5f09b" (UID: "a5f5b015-9b45-407f-ae2b-9d9645b5f09b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:24.985563 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:24.985514    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f5b015-9b45-407f-ae2b-9d9645b5f09b-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:25.230829 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:25.230774    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc838bd-671f-463d-b98e-0435f56ff70e" path="/var/lib/kubelet/pods/6bc838bd-671f-463d-b98e-0435f56ff70e/volumes"
Oct 29 22:49:25.429124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:25.429098    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"
Oct 29 22:49:25.429580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:25.429130    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq" event={"ID":"a5f5b015-9b45-407f-ae2b-9d9645b5f09b","Type":"ContainerDied","Data":"e622ec996ed6ee70047e9ea7e31cf1448e932d232b88540e175e1300765469d3"}
Oct 29 22:49:25.429580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:25.429205    4559 scope.go:122] "RemoveContainer" containerID="9ea5448f1da7cee51107198c0864e68bae1e77652b15d621f0599d28ac0071b7"
Oct 29 22:49:25.509458 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:25.509420    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"]
Oct 29 22:49:25.517487 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:25.517457    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5a69c9-7779f4d998-xrblq"]
Oct 29 22:49:25.570010 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:25.569839    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 29 22:49:25.570010 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:25.569910    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq" podUID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 29 22:49:26.376839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.376333    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"]
Oct 29 22:49:26.377775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.377661    4559 cpu_manager.go:414] "containerMap: removing container" podUID="702eed95-27cd-42a7-a7fb-f14bcc951474" containerName="istio-proxy"
Oct 29 22:49:26.377775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.377687    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702eed95-27cd-42a7-a7fb-f14bcc951474" containerName="istio-proxy"
Oct 29 22:49:26.377775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.377701    4559 cpu_manager.go:414] "containerMap: removing container" podUID="8c044857-200a-41c6-bbd3-e3f35bf44dc3" containerName="istio-proxy"
Oct 29 22:49:26.377775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.377709    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c044857-200a-41c6-bbd3-e3f35bf44dc3" containerName="istio-proxy"
Oct 29 22:49:26.377775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.377747    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a5f5b015-9b45-407f-ae2b-9d9645b5f09b" containerName="payload-processing"
Oct 29 22:49:26.377775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.377756    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f5b015-9b45-407f-ae2b-9d9645b5f09b" containerName="payload-processing"
Oct 29 22:49:26.377775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.377780    4559 cpu_manager.go:414] "containerMap: removing container" podUID="5a29c9ee-f1a8-460a-a4b7-25625aa72bfd" containerName="payload-processing"
Oct 29 22:49:26.377775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.377787    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a29c9ee-f1a8-460a-a4b7-25625aa72bfd" containerName="payload-processing"
Oct 29 22:49:26.377775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.377802    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6bc838bd-671f-463d-b98e-0435f56ff70e" containerName="istio-proxy"
Oct 29 22:49:26.377775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.377824    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc838bd-671f-463d-b98e-0435f56ff70e" containerName="istio-proxy"
Oct 29 22:49:26.377775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.377837    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e9e758b8-4105-41d2-a2f6-49e874789dfb" containerName="istio-proxy"
Oct 29 22:49:26.377775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.377844    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e758b8-4105-41d2-a2f6-49e874789dfb" containerName="istio-proxy"
Oct 29 22:49:26.378513 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.378023    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="702eed95-27cd-42a7-a7fb-f14bcc951474" containerName="istio-proxy"
Oct 29 22:49:26.378513 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.378037    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c044857-200a-41c6-bbd3-e3f35bf44dc3" containerName="istio-proxy"
Oct 29 22:49:26.378513 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.378052    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f5b015-9b45-407f-ae2b-9d9645b5f09b" containerName="payload-processing"
Oct 29 22:49:26.378513 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.378064    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a29c9ee-f1a8-460a-a4b7-25625aa72bfd" containerName="payload-processing"
Oct 29 22:49:26.378513 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.378075    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc838bd-671f-463d-b98e-0435f56ff70e" containerName="istio-proxy"
Oct 29 22:49:26.378513 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.378084    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e758b8-4105-41d2-a2f6-49e874789dfb" containerName="istio-proxy"
Oct 29 22:49:26.387259 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.385613    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.397188 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.397146    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"]
Oct 29 22:49:26.404329 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.404304    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-c933cb-openshift-default-dockercfg-f99fc\""
Oct 29 22:49:26.412918 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.412895    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-envoy\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.413146 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.413123    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-credential-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.413297 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.413274    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-token\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.413422 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.413410    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4pkb\" (UniqueName: \"kubernetes.io/projected/b905ef8e-61e8-4693-ae47-1b3940e87b0d-kube-api-access-r4pkb\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.413527 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.413516    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-podinfo\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.413692 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.413681    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-workload-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.413841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.413830    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-data\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.413971 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.413957    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-workload-certs\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.414129 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.414118    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istiod-ca-cert\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.549851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.543586    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4pkb\" (UniqueName: \"kubernetes.io/projected/b905ef8e-61e8-4693-ae47-1b3940e87b0d-kube-api-access-r4pkb\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.549851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.543651    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-podinfo\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.549851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.543754    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-workload-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.549851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.543847    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-data\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.549851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.543898    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-workload-certs\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.549851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.543997    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istiod-ca-cert\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.549851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.544072    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-envoy\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.549851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.544136    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-credential-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.549851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.544221    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-token\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.707901 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.621247    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 29 22:49:26.707901 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.621319    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq" podUID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 29 22:49:26.707901 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.662758    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-credential-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.707901 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.697244    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-podinfo\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.746644 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.708458    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-workload-certs\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.746644 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.708791    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-data\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.746644 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.716376    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istiod-ca-cert\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.746992 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.746689    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-workload-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.773837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.770363    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-token\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.785517 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.785010    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-envoy\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:26.849842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:26.846568    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4pkb\" (UniqueName: \"kubernetes.io/projected/b905ef8e-61e8-4693-ae47-1b3940e87b0d-kube-api-access-r4pkb\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:27.016013 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:27.015918    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:27.575853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:27.569630    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 29 22:49:27.575853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:27.569715    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq" podUID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 29 22:49:27.780768 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:27.780731    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f5b015-9b45-407f-ae2b-9d9645b5f09b" path="/var/lib/kubelet/pods/a5f5b015-9b45-407f-ae2b-9d9645b5f09b/volumes"
Oct 29 22:49:27.781757 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:27.781725    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"]
Oct 29 22:49:27.790189 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:27.789672    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"]
Oct 29 22:49:27.791964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:27.791921    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:49:27.792074 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:27.792014    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:49:27.792074 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:27.792064    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:49:27.797986 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:27.797866    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"]
Oct 29 22:49:27.811990 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:27.811953    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84" event={"ID":"b905ef8e-61e8-4693-ae47-1b3940e87b0d","Type":"ContainerStarted","Data":"b5d8de0502d8bf15c01cba1aef3453df002907d8de217a67efee9a6a07f8d612"}
Oct 29 22:49:28.345892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:28.344929    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:49:28.571841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:28.569261    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 29 22:49:28.571841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:28.569324    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq" podUID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 29 22:49:28.824517 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:28.824472    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84" event={"ID":"b905ef8e-61e8-4693-ae47-1b3940e87b0d","Type":"ContainerStarted","Data":"6345299c7471ecfc26af9d2495cc7bcddb76905dcdefb020bb29f8558b74e683"}
Oct 29 22:49:28.883770 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:28.883707    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84" podStartSLOduration=2.88369235 podStartE2EDuration="2.88369235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:28.879076481 +0000 UTC m=+2107.790520071" watchObservedRunningTime="2026-10-29 22:49:28.88369235 +0000 UTC m=+2107.795135963"
Oct 29 22:49:29.030842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:29.024014    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:29.087945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:29.087857    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:29.439935 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:29.439901    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:49:29.574180 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:29.574148    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:49:29.575408 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:29.575386    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 22:49:29.970347 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:29.970313    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:29.971430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:29.971100    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:49:32.309469 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:32.309436    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"
Oct 29 22:49:32.415570 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:32.415539    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/068d019b-57d0-4974-9aba-aab57e23b4fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/068d019b-57d0-4974-9aba-aab57e23b4fb-plugins-config-volume\") pod \"068d019b-57d0-4974-9aba-aab57e23b4fb\" (UID: \"068d019b-57d0-4974-9aba-aab57e23b4fb\") "
Oct 29 22:49:32.415867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:32.415850    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068d019b-57d0-4974-9aba-aab57e23b4fb-kube-api-access-j2npj\" (UniqueName: \"kubernetes.io/projected/068d019b-57d0-4974-9aba-aab57e23b4fb-kube-api-access-j2npj\") pod \"068d019b-57d0-4974-9aba-aab57e23b4fb\" (UID: \"068d019b-57d0-4974-9aba-aab57e23b4fb\") "
Oct 29 22:49:32.415999 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:32.415974    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068d019b-57d0-4974-9aba-aab57e23b4fb-plugins-config-volume" pod "068d019b-57d0-4974-9aba-aab57e23b4fb" (UID: "068d019b-57d0-4974-9aba-aab57e23b4fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:32.416321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:32.416300    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/068d019b-57d0-4974-9aba-aab57e23b4fb-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:32.418137 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:32.418115    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068d019b-57d0-4974-9aba-aab57e23b4fb-kube-api-access-j2npj" pod "068d019b-57d0-4974-9aba-aab57e23b4fb" (UID: "068d019b-57d0-4974-9aba-aab57e23b4fb"). InnerVolumeSpecName "kube-api-access-j2npj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:32.517872 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:32.517800    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2npj\" (UniqueName: \"kubernetes.io/projected/068d019b-57d0-4974-9aba-aab57e23b4fb-kube-api-access-j2npj\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:32.782294 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:32.782246    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv" podUID="b19e1fbe-b078-4a6b-a2c0-c30f09df2962" containerName="payload-processing" containerID="cri-o://b37f46a6ab9ce78d5a80188de18ac7b3a3755b3b9f0832800c8b5945a4a7b8fd" gracePeriod=25
Oct 29 22:49:32.790454 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:32.790409    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v" podUID="a4109193-5cc4-4dcb-9f8d-fbbe5e359010" containerName="payload-pre-processing" containerID="cri-o://d69054a2db02d913431abd772f1ed9f865efd4bc1a2b8659adb8f86ec6625d06" gracePeriod=25
Oct 29 22:49:33.028430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.028389    4559 generic.go:356] "Generic (PLEG): container finished" podID="068d019b-57d0-4974-9aba-aab57e23b4fb" containerID="4c1d88065c83286b5a0dc5ba5c747bdc3922fca7030a10c6c96edf690d0a42cd" exitCode=0
Oct 29 22:49:33.028646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.028519    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp" event={"ID":"068d019b-57d0-4974-9aba-aab57e23b4fb","Type":"ContainerDied","Data":"4c1d88065c83286b5a0dc5ba5c747bdc3922fca7030a10c6c96edf690d0a42cd"}
Oct 29 22:49:33.028646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.028595    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp" event={"ID":"068d019b-57d0-4974-9aba-aab57e23b4fb","Type":"ContainerDied","Data":"4d71c4f18a394a21ba38b8634257f28a5636059099ab37e65488e36a685c494f"}
Oct 29 22:49:33.028646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.028629    4559 scope.go:122] "RemoveContainer" containerID="4c1d88065c83286b5a0dc5ba5c747bdc3922fca7030a10c6c96edf690d0a42cd"
Oct 29 22:49:33.028900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.028880    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"
Oct 29 22:49:33.039374 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.039035    4559 generic.go:356] "Generic (PLEG): container finished" podID="b19e1fbe-b078-4a6b-a2c0-c30f09df2962" containerID="b37f46a6ab9ce78d5a80188de18ac7b3a3755b3b9f0832800c8b5945a4a7b8fd" exitCode=0
Oct 29 22:49:33.039374 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.039306    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv" event={"ID":"b19e1fbe-b078-4a6b-a2c0-c30f09df2962","Type":"ContainerDied","Data":"b37f46a6ab9ce78d5a80188de18ac7b3a3755b3b9f0832800c8b5945a4a7b8fd"}
Oct 29 22:49:33.059430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.059404    4559 scope.go:122] "RemoveContainer" containerID="4c1d88065c83286b5a0dc5ba5c747bdc3922fca7030a10c6c96edf690d0a42cd"
Oct 29 22:49:33.059901 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:33.059869    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c1d88065c83286b5a0dc5ba5c747bdc3922fca7030a10c6c96edf690d0a42cd\": container with ID starting with 4c1d88065c83286b5a0dc5ba5c747bdc3922fca7030a10c6c96edf690d0a42cd not found: ID does not exist" containerID="4c1d88065c83286b5a0dc5ba5c747bdc3922fca7030a10c6c96edf690d0a42cd"
Oct 29 22:49:33.060024 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.059907    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c1d88065c83286b5a0dc5ba5c747bdc3922fca7030a10c6c96edf690d0a42cd"} err="failed to get container status \"4c1d88065c83286b5a0dc5ba5c747bdc3922fca7030a10c6c96edf690d0a42cd\": rpc error: code = NotFound desc = could not find container \"4c1d88065c83286b5a0dc5ba5c747bdc3922fca7030a10c6c96edf690d0a42cd\": container with ID starting with 4c1d88065c83286b5a0dc5ba5c747bdc3922fca7030a10c6c96edf690d0a42cd not found: ID does not exist"
Oct 29 22:49:33.101163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.101136    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"]
Oct 29 22:49:33.107423 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.107384    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"
Oct 29 22:49:33.119124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.119091    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c7d192-fcf758f79-fd7hp"]
Oct 29 22:49:33.230829 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.230791    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b19e1fbe-b078-4a6b-a2c0-c30f09df2962-kube-api-access-hnl7n\" (UniqueName: \"kubernetes.io/projected/b19e1fbe-b078-4a6b-a2c0-c30f09df2962-kube-api-access-hnl7n\") pod \"b19e1fbe-b078-4a6b-a2c0-c30f09df2962\" (UID: \"b19e1fbe-b078-4a6b-a2c0-c30f09df2962\") "
Oct 29 22:49:33.231060 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.231040    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b19e1fbe-b078-4a6b-a2c0-c30f09df2962-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b19e1fbe-b078-4a6b-a2c0-c30f09df2962-plugins-config-volume\") pod \"b19e1fbe-b078-4a6b-a2c0-c30f09df2962\" (UID: \"b19e1fbe-b078-4a6b-a2c0-c30f09df2962\") "
Oct 29 22:49:33.231479 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.231445    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b19e1fbe-b078-4a6b-a2c0-c30f09df2962-plugins-config-volume" pod "b19e1fbe-b078-4a6b-a2c0-c30f09df2962" (UID: "b19e1fbe-b078-4a6b-a2c0-c30f09df2962"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:33.231895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.231873    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b19e1fbe-b078-4a6b-a2c0-c30f09df2962-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:33.232524 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.232498    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="068d019b-57d0-4974-9aba-aab57e23b4fb" path="/var/lib/kubelet/pods/068d019b-57d0-4974-9aba-aab57e23b4fb/volumes"
Oct 29 22:49:33.233291 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.233269    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19e1fbe-b078-4a6b-a2c0-c30f09df2962-kube-api-access-hnl7n" pod "b19e1fbe-b078-4a6b-a2c0-c30f09df2962" (UID: "b19e1fbe-b078-4a6b-a2c0-c30f09df2962"). InnerVolumeSpecName "kube-api-access-hnl7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:33.326191 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.326113    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"]
Oct 29 22:49:33.326880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.326863    4559 cpu_manager.go:414] "containerMap: removing container" podUID="068d019b-57d0-4974-9aba-aab57e23b4fb" containerName="payload-pre-processing"
Oct 29 22:49:33.326880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.326880    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068d019b-57d0-4974-9aba-aab57e23b4fb" containerName="payload-pre-processing"
Oct 29 22:49:33.327018 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.326904    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b19e1fbe-b078-4a6b-a2c0-c30f09df2962" containerName="payload-processing"
Oct 29 22:49:33.327018 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.326910    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b19e1fbe-b078-4a6b-a2c0-c30f09df2962" containerName="payload-processing"
Oct 29 22:49:33.327087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.327033    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="068d019b-57d0-4974-9aba-aab57e23b4fb" containerName="payload-pre-processing"
Oct 29 22:49:33.327087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.327042    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b19e1fbe-b078-4a6b-a2c0-c30f09df2962" containerName="payload-processing"
Oct 29 22:49:33.334228 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.334197    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnl7n\" (UniqueName: \"kubernetes.io/projected/b19e1fbe-b078-4a6b-a2c0-c30f09df2962-kube-api-access-hnl7n\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:33.334957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.334934    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"
Oct 29 22:49:33.352678 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.352000    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-c933cb\""
Oct 29 22:49:33.352678 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.352514    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-c933cb-dockercfg-wmrd6\""
Oct 29 22:49:33.356131 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.356100    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"]
Oct 29 22:49:33.635976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.635258    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5xf2\" (UniqueName: \"kubernetes.io/projected/2b2f4588-6420-4161-a8f1-f7a888ff1993-kube-api-access-h5xf2\") pod \"payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg\" (UID: \"2b2f4588-6420-4161-a8f1-f7a888ff1993\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"
Oct 29 22:49:33.635976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.635428    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2f4588-6420-4161-a8f1-f7a888ff1993-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg\" (UID: \"2b2f4588-6420-4161-a8f1-f7a888ff1993\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"
Oct 29 22:49:33.739843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.737367    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5xf2\" (UniqueName: \"kubernetes.io/projected/2b2f4588-6420-4161-a8f1-f7a888ff1993-kube-api-access-h5xf2\") pod \"payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg\" (UID: \"2b2f4588-6420-4161-a8f1-f7a888ff1993\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"
Oct 29 22:49:33.739843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.737527    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2f4588-6420-4161-a8f1-f7a888ff1993-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg\" (UID: \"2b2f4588-6420-4161-a8f1-f7a888ff1993\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"
Oct 29 22:49:33.739843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.738727    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2f4588-6420-4161-a8f1-f7a888ff1993-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg\" (UID: \"2b2f4588-6420-4161-a8f1-f7a888ff1993\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"
Oct 29 22:49:33.761834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.741044    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"]
Oct 29 22:49:33.812268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.812218    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"]
Oct 29 22:49:33.812508 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.812367    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"
Oct 29 22:49:33.818925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.815123    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5xf2\" (UniqueName: \"kubernetes.io/projected/2b2f4588-6420-4161-a8f1-f7a888ff1993-kube-api-access-h5xf2\") pod \"payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg\" (UID: \"2b2f4588-6420-4161-a8f1-f7a888ff1993\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"
Oct 29 22:49:33.838894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.838858    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85956\" (UniqueName: \"kubernetes.io/projected/11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24-kube-api-access-85956\") pod \"payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z\" (UID: \"11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"
Oct 29 22:49:33.839072 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.839032    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z\" (UID: \"11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"
Oct 29 22:49:33.940865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.940824    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85956\" (UniqueName: \"kubernetes.io/projected/11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24-kube-api-access-85956\") pod \"payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z\" (UID: \"11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"
Oct 29 22:49:33.941082 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.940930    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z\" (UID: \"11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"
Oct 29 22:49:33.941668 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.941638    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z\" (UID: \"11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"
Oct 29 22:49:33.951780 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.951750    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"
Oct 29 22:49:33.960989 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:33.960895    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85956\" (UniqueName: \"kubernetes.io/projected/11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24-kube-api-access-85956\") pod \"payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z\" (UID: \"11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"
Oct 29 22:49:34.056232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:34.056197    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv" event={"ID":"b19e1fbe-b078-4a6b-a2c0-c30f09df2962","Type":"ContainerDied","Data":"2137b71526f8ee013a579d9c327db99162d7688bcedf97579a61eed82492a772"}
Oct 29 22:49:34.057844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:34.056984    4559 scope.go:122] "RemoveContainer" containerID="b37f46a6ab9ce78d5a80188de18ac7b3a3755b3b9f0832800c8b5945a4a7b8fd"
Oct 29 22:49:34.057844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:34.056682    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"
Oct 29 22:49:34.119438 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:34.119400    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"]
Oct 29 22:49:34.123696 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:34.123667    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a3ef0-9d775496-fs4gv"]
Oct 29 22:49:34.148494 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:34.148454    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"
Oct 29 22:49:34.264147 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:49:34.260366    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2f4588_6420_4161_a8f1_f7a888ff1993.slice/crio-e06b9364c9442dea250d8dfdb6a28511032fc11b140a03727fc2c1a9b9634458 WatchSource:0}: Error finding container e06b9364c9442dea250d8dfdb6a28511032fc11b140a03727fc2c1a9b9634458: Status 404 returned error can't find the container with id e06b9364c9442dea250d8dfdb6a28511032fc11b140a03727fc2c1a9b9634458
Oct 29 22:49:34.280552 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:34.280519    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"]
Oct 29 22:49:34.458424 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:49:34.458347    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c8ba74_8c08_4c3e_a6ac_59cdf7b88d24.slice/crio-7f647527d2b7e063876839ccd6f75f45ec64bfe523af9359ee097a72d8daae38 WatchSource:0}: Error finding container 7f647527d2b7e063876839ccd6f75f45ec64bfe523af9359ee097a72d8daae38: Status 404 returned error can't find the container with id 7f647527d2b7e063876839ccd6f75f45ec64bfe523af9359ee097a72d8daae38
Oct 29 22:49:34.458424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:34.458384    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"]
Oct 29 22:49:35.069294 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:35.069251    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg" event={"ID":"2b2f4588-6420-4161-a8f1-f7a888ff1993","Type":"ContainerStarted","Data":"a3f8df9d925f6459ed1d048e28ee3603a78a1727d6509bd2eb98cd46806fd44a"}
Oct 29 22:49:35.070917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:35.069320    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg" event={"ID":"2b2f4588-6420-4161-a8f1-f7a888ff1993","Type":"ContainerStarted","Data":"e06b9364c9442dea250d8dfdb6a28511032fc11b140a03727fc2c1a9b9634458"}
Oct 29 22:49:35.070917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:35.069450    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"
Oct 29 22:49:35.072166 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:35.072140    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z" event={"ID":"11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24","Type":"ContainerStarted","Data":"acbc176d1b91b36a2279837f7589acc4c0d3c18de463405ac1730b4c5deb7a99"}
Oct 29 22:49:35.072287 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:35.072191    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z" event={"ID":"11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24","Type":"ContainerStarted","Data":"7f647527d2b7e063876839ccd6f75f45ec64bfe523af9359ee097a72d8daae38"}
Oct 29 22:49:35.072287 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:35.072257    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"
Oct 29 22:49:35.118831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:35.118748    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg" podStartSLOduration=2.118729188 podStartE2EDuration="2.118729188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:35.116099381 +0000 UTC m=+2114.027542937" watchObservedRunningTime="2026-10-29 22:49:35.118729188 +0000 UTC m=+2114.030172743"
Oct 29 22:49:35.128512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:35.128476    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"]
Oct 29 22:49:35.129016 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:35.128979    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh" podUID="b6cffbad-4429-4e61-b582-5b45fda07b48" containerName="istio-proxy" containerID="cri-o://1d4f576a2dabe48747affe92fdca5d08de1f6ecbba49e7fdce2e3da1d56690c0" gracePeriod=30
Oct 29 22:49:35.162702 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:35.157911    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z" podStartSLOduration=2.157896663 podStartE2EDuration="2.157896663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:35.156157719 +0000 UTC m=+2114.067601273" watchObservedRunningTime="2026-10-29 22:49:35.157896663 +0000 UTC m=+2114.069340218"
Oct 29 22:49:35.229251 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:35.229210    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b19e1fbe-b078-4a6b-a2c0-c30f09df2962" path="/var/lib/kubelet/pods/b19e1fbe-b078-4a6b-a2c0-c30f09df2962/volumes"
Oct 29 22:49:36.492407 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:36.492368    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v" podUID="a4109193-5cc4-4dcb-9f8d-fbbe5e359010" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:40.478561 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.478524    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:49:40.590709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.587325    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-podinfo\") pod \"b6cffbad-4429-4e61-b582-5b45fda07b48\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") "
Oct 29 22:49:40.590709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.587434    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-envoy\") pod \"b6cffbad-4429-4e61-b582-5b45fda07b48\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") "
Oct 29 22:49:40.590709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.587552    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-credential-socket\") pod \"b6cffbad-4429-4e61-b582-5b45fda07b48\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") "
Oct 29 22:49:40.590709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.587589    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-workload-certs\") pod \"b6cffbad-4429-4e61-b582-5b45fda07b48\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") "
Oct 29 22:49:40.590709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.587621    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-data\") pod \"b6cffbad-4429-4e61-b582-5b45fda07b48\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") "
Oct 29 22:49:40.590709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.587683    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6cffbad-4429-4e61-b582-5b45fda07b48-kube-api-access-gpxbv\" (UniqueName: \"kubernetes.io/projected/b6cffbad-4429-4e61-b582-5b45fda07b48-kube-api-access-gpxbv\") pod \"b6cffbad-4429-4e61-b582-5b45fda07b48\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") "
Oct 29 22:49:40.590709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.587753    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-workload-socket\") pod \"b6cffbad-4429-4e61-b582-5b45fda07b48\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") "
Oct 29 22:49:40.590709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.587785    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-token\" (UniqueName: \"kubernetes.io/projected/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-token\") pod \"b6cffbad-4429-4e61-b582-5b45fda07b48\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") "
Oct 29 22:49:40.590709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.588042    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6cffbad-4429-4e61-b582-5b45fda07b48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cffbad-4429-4e61-b582-5b45fda07b48-istiod-ca-cert\") pod \"b6cffbad-4429-4e61-b582-5b45fda07b48\" (UID: \"b6cffbad-4429-4e61-b582-5b45fda07b48\") "
Oct 29 22:49:40.632082 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.632036    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6cffbad-4429-4e61-b582-5b45fda07b48-istiod-ca-cert" pod "b6cffbad-4429-4e61-b582-5b45fda07b48" (UID: "b6cffbad-4429-4e61-b582-5b45fda07b48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:40.652867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.642451    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-credential-socket" pod "b6cffbad-4429-4e61-b582-5b45fda07b48" (UID: "b6cffbad-4429-4e61-b582-5b45fda07b48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:40.652867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.642714    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-workload-certs" pod "b6cffbad-4429-4e61-b582-5b45fda07b48" (UID: "b6cffbad-4429-4e61-b582-5b45fda07b48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:40.652867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.645860    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-envoy" pod "b6cffbad-4429-4e61-b582-5b45fda07b48" (UID: "b6cffbad-4429-4e61-b582-5b45fda07b48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:40.652867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.646085    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-data" pod "b6cffbad-4429-4e61-b582-5b45fda07b48" (UID: "b6cffbad-4429-4e61-b582-5b45fda07b48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:40.652867 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.646269    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-workload-socket" pod "b6cffbad-4429-4e61-b582-5b45fda07b48" (UID: "b6cffbad-4429-4e61-b582-5b45fda07b48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:40.656113 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.653581    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-podinfo" pod "b6cffbad-4429-4e61-b582-5b45fda07b48" (UID: "b6cffbad-4429-4e61-b582-5b45fda07b48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:40.660026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.659994    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cffbad-4429-4e61-b582-5b45fda07b48-kube-api-access-gpxbv" pod "b6cffbad-4429-4e61-b582-5b45fda07b48" (UID: "b6cffbad-4429-4e61-b582-5b45fda07b48"). InnerVolumeSpecName "kube-api-access-gpxbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:40.660229 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.660163    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-token" pod "b6cffbad-4429-4e61-b582-5b45fda07b48" (UID: "b6cffbad-4429-4e61-b582-5b45fda07b48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:40.693891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.690271    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cffbad-4429-4e61-b582-5b45fda07b48-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.693891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.690311    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.693891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.690324    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.693891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.690338    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.693891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.690354    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.693891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.690368    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.693891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.690380    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpxbv\" (UniqueName: \"kubernetes.io/projected/b6cffbad-4429-4e61-b582-5b45fda07b48-kube-api-access-gpxbv\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.693891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.690395    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cffbad-4429-4e61-b582-5b45fda07b48-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.693891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:40.690408    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cffbad-4429-4e61-b582-5b45fda07b48-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:41.187341 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:41.187285    4559 generic.go:356] "Generic (PLEG): container finished" podID="b6cffbad-4429-4e61-b582-5b45fda07b48" containerID="1d4f576a2dabe48747affe92fdca5d08de1f6ecbba49e7fdce2e3da1d56690c0" exitCode=0
Oct 29 22:49:41.187341 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:41.187341    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"
Oct 29 22:49:41.187584 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:41.187355    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh" event={"ID":"b6cffbad-4429-4e61-b582-5b45fda07b48","Type":"ContainerDied","Data":"1d4f576a2dabe48747affe92fdca5d08de1f6ecbba49e7fdce2e3da1d56690c0"}
Oct 29 22:49:41.187584 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:41.187420    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh" event={"ID":"b6cffbad-4429-4e61-b582-5b45fda07b48","Type":"ContainerDied","Data":"b6c9bc96d5a0b793fc7325bc8b79a46c366dc43d2b3c15fe06879a3b91671da7"}
Oct 29 22:49:41.187584 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:41.187452    4559 scope.go:122] "RemoveContainer" containerID="1d4f576a2dabe48747affe92fdca5d08de1f6ecbba49e7fdce2e3da1d56690c0"
Oct 29 22:49:41.200954 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:41.200793    4559 scope.go:122] "RemoveContainer" containerID="11457e1ec66e14af1203fc2fe0d0bd584389b64c10606d5266cef0f8a5f31ffb"
Oct 29 22:49:41.213994 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:41.213972    4559 scope.go:122] "RemoveContainer" containerID="1d4f576a2dabe48747affe92fdca5d08de1f6ecbba49e7fdce2e3da1d56690c0"
Oct 29 22:49:41.214383 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:41.214359    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d4f576a2dabe48747affe92fdca5d08de1f6ecbba49e7fdce2e3da1d56690c0\": container with ID starting with 1d4f576a2dabe48747affe92fdca5d08de1f6ecbba49e7fdce2e3da1d56690c0 not found: ID does not exist" containerID="1d4f576a2dabe48747affe92fdca5d08de1f6ecbba49e7fdce2e3da1d56690c0"
Oct 29 22:49:41.214488 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:41.214392    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d4f576a2dabe48747affe92fdca5d08de1f6ecbba49e7fdce2e3da1d56690c0"} err="failed to get container status \"1d4f576a2dabe48747affe92fdca5d08de1f6ecbba49e7fdce2e3da1d56690c0\": rpc error: code = NotFound desc = could not find container \"1d4f576a2dabe48747affe92fdca5d08de1f6ecbba49e7fdce2e3da1d56690c0\": container with ID starting with 1d4f576a2dabe48747affe92fdca5d08de1f6ecbba49e7fdce2e3da1d56690c0 not found: ID does not exist"
Oct 29 22:49:41.214488 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:41.214409    4559 scope.go:122] "RemoveContainer" containerID="11457e1ec66e14af1203fc2fe0d0bd584389b64c10606d5266cef0f8a5f31ffb"
Oct 29 22:49:41.214704 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:41.214682    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11457e1ec66e14af1203fc2fe0d0bd584389b64c10606d5266cef0f8a5f31ffb\": container with ID starting with 11457e1ec66e14af1203fc2fe0d0bd584389b64c10606d5266cef0f8a5f31ffb not found: ID does not exist" containerID="11457e1ec66e14af1203fc2fe0d0bd584389b64c10606d5266cef0f8a5f31ffb"
Oct 29 22:49:41.214760 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:41.214715    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11457e1ec66e14af1203fc2fe0d0bd584389b64c10606d5266cef0f8a5f31ffb"} err="failed to get container status \"11457e1ec66e14af1203fc2fe0d0bd584389b64c10606d5266cef0f8a5f31ffb\": rpc error: code = NotFound desc = could not find container \"11457e1ec66e14af1203fc2fe0d0bd584389b64c10606d5266cef0f8a5f31ffb\": container with ID starting with 11457e1ec66e14af1203fc2fe0d0bd584389b64c10606d5266cef0f8a5f31ffb not found: ID does not exist"
Oct 29 22:49:41.264200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:41.264163    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"]
Oct 29 22:49:41.274211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:41.274174    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-bfffh"]
Oct 29 22:49:43.229439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:43.229404    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6cffbad-4429-4e61-b582-5b45fda07b48" path="/var/lib/kubelet/pods/b6cffbad-4429-4e61-b582-5b45fda07b48/volumes"
Oct 29 22:49:43.696788 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:43.696752    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:49:45.855411 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:45.855372    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"]
Oct 29 22:49:46.086852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:46.086796    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"
Oct 29 22:49:46.087481 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:46.087460    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"
Oct 29 22:49:46.493783 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:46.493740    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v" podUID="a4109193-5cc4-4dcb-9f8d-fbbe5e359010" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:48.124645 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:48.124606    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"]
Oct 29 22:49:48.125609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:48.125574    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd" podUID="44273e03-eb0c-48e0-82ed-9adfc7badd3d" containerName="istio-proxy" containerID="cri-o://bb7959c54cc9c8a0b46975390a69787da7d8237ed40acb14a3f238c01ac27d39" gracePeriod=30
Oct 29 22:49:50.855888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:50.855840    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4" podUID="02cf8b24-105b-4675-a4de-8c8c77e4afa9" containerName="payload-processing" containerID="cri-o://31959f840e48a0b205d094e363cd6d9c6c44aadc8c280711d0f2e02b86157326" gracePeriod=25
Oct 29 22:49:51.148897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.148862    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"
Oct 29 22:49:51.210629 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.210596    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02cf8b24-105b-4675-a4de-8c8c77e4afa9-kube-api-access-rwdhf\" (UniqueName: \"kubernetes.io/projected/02cf8b24-105b-4675-a4de-8c8c77e4afa9-kube-api-access-rwdhf\") pod \"02cf8b24-105b-4675-a4de-8c8c77e4afa9\" (UID: \"02cf8b24-105b-4675-a4de-8c8c77e4afa9\") "
Oct 29 22:49:51.210798 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.210763    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02cf8b24-105b-4675-a4de-8c8c77e4afa9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cf8b24-105b-4675-a4de-8c8c77e4afa9-plugins-config-volume\") pod \"02cf8b24-105b-4675-a4de-8c8c77e4afa9\" (UID: \"02cf8b24-105b-4675-a4de-8c8c77e4afa9\") "
Oct 29 22:49:51.211234 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.211204    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02cf8b24-105b-4675-a4de-8c8c77e4afa9-plugins-config-volume" pod "02cf8b24-105b-4675-a4de-8c8c77e4afa9" (UID: "02cf8b24-105b-4675-a4de-8c8c77e4afa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:51.211590 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.211524    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cf8b24-105b-4675-a4de-8c8c77e4afa9-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:51.213268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.213237    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02cf8b24-105b-4675-a4de-8c8c77e4afa9-kube-api-access-rwdhf" pod "02cf8b24-105b-4675-a4de-8c8c77e4afa9" (UID: "02cf8b24-105b-4675-a4de-8c8c77e4afa9"). InnerVolumeSpecName "kube-api-access-rwdhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:51.272708 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.272675    4559 generic.go:356] "Generic (PLEG): container finished" podID="02cf8b24-105b-4675-a4de-8c8c77e4afa9" containerID="31959f840e48a0b205d094e363cd6d9c6c44aadc8c280711d0f2e02b86157326" exitCode=0
Oct 29 22:49:51.272918 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.272724    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"
Oct 29 22:49:51.272918 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.272768    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4" event={"ID":"02cf8b24-105b-4675-a4de-8c8c77e4afa9","Type":"ContainerDied","Data":"31959f840e48a0b205d094e363cd6d9c6c44aadc8c280711d0f2e02b86157326"}
Oct 29 22:49:51.272918 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.272843    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4" event={"ID":"02cf8b24-105b-4675-a4de-8c8c77e4afa9","Type":"ContainerDied","Data":"d2871d8061d02d663158acc201ba5f6d4830a292f71ab3e68ab76e1892dafc2b"}
Oct 29 22:49:51.272918 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.272870    4559 scope.go:122] "RemoveContainer" containerID="31959f840e48a0b205d094e363cd6d9c6c44aadc8c280711d0f2e02b86157326"
Oct 29 22:49:51.284030 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.284009    4559 scope.go:122] "RemoveContainer" containerID="31959f840e48a0b205d094e363cd6d9c6c44aadc8c280711d0f2e02b86157326"
Oct 29 22:49:51.284348 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:51.284326    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31959f840e48a0b205d094e363cd6d9c6c44aadc8c280711d0f2e02b86157326\": container with ID starting with 31959f840e48a0b205d094e363cd6d9c6c44aadc8c280711d0f2e02b86157326 not found: ID does not exist" containerID="31959f840e48a0b205d094e363cd6d9c6c44aadc8c280711d0f2e02b86157326"
Oct 29 22:49:51.284438 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.284359    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31959f840e48a0b205d094e363cd6d9c6c44aadc8c280711d0f2e02b86157326"} err="failed to get container status \"31959f840e48a0b205d094e363cd6d9c6c44aadc8c280711d0f2e02b86157326\": rpc error: code = NotFound desc = could not find container \"31959f840e48a0b205d094e363cd6d9c6c44aadc8c280711d0f2e02b86157326\": container with ID starting with 31959f840e48a0b205d094e363cd6d9c6c44aadc8c280711d0f2e02b86157326 not found: ID does not exist"
Oct 29 22:49:51.313400 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.313365    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwdhf\" (UniqueName: \"kubernetes.io/projected/02cf8b24-105b-4675-a4de-8c8c77e4afa9-kube-api-access-rwdhf\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:51.334519 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.334482    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"]
Oct 29 22:49:51.340011 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.339981    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-05417d-754c7b55dd-kclz4"]
Oct 29 22:49:51.587730 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.587692    4559 patch_prober.go:29] interesting pod/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:49:51.587954 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:51.587750    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd" podUID="44273e03-eb0c-48e0-82ed-9adfc7badd3d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:49:53.126523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.126490    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"
Oct 29 22:49:53.201823 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.201765    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4109193-5cc4-4dcb-9f8d-fbbe5e359010-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4109193-5cc4-4dcb-9f8d-fbbe5e359010-plugins-config-volume\") pod \"a4109193-5cc4-4dcb-9f8d-fbbe5e359010\" (UID: \"a4109193-5cc4-4dcb-9f8d-fbbe5e359010\") "
Oct 29 22:49:53.202020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.201870    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4109193-5cc4-4dcb-9f8d-fbbe5e359010-kube-api-access-wz9kg\" (UniqueName: \"kubernetes.io/projected/a4109193-5cc4-4dcb-9f8d-fbbe5e359010-kube-api-access-wz9kg\") pod \"a4109193-5cc4-4dcb-9f8d-fbbe5e359010\" (UID: \"a4109193-5cc4-4dcb-9f8d-fbbe5e359010\") "
Oct 29 22:49:53.202268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.202237    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4109193-5cc4-4dcb-9f8d-fbbe5e359010-plugins-config-volume" pod "a4109193-5cc4-4dcb-9f8d-fbbe5e359010" (UID: "a4109193-5cc4-4dcb-9f8d-fbbe5e359010"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:53.202696 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.202660    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4109193-5cc4-4dcb-9f8d-fbbe5e359010-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.204878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.204847    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4109193-5cc4-4dcb-9f8d-fbbe5e359010-kube-api-access-wz9kg" pod "a4109193-5cc4-4dcb-9f8d-fbbe5e359010" (UID: "a4109193-5cc4-4dcb-9f8d-fbbe5e359010"). InnerVolumeSpecName "kube-api-access-wz9kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:53.229274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.229239    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02cf8b24-105b-4675-a4de-8c8c77e4afa9" path="/var/lib/kubelet/pods/02cf8b24-105b-4675-a4de-8c8c77e4afa9/volumes"
Oct 29 22:49:53.331851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.318167    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz9kg\" (UniqueName: \"kubernetes.io/projected/a4109193-5cc4-4dcb-9f8d-fbbe5e359010-kube-api-access-wz9kg\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.390219 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.387471    4559 generic.go:356] "Generic (PLEG): container finished" podID="44273e03-eb0c-48e0-82ed-9adfc7badd3d" containerID="bb7959c54cc9c8a0b46975390a69787da7d8237ed40acb14a3f238c01ac27d39" exitCode=0
Oct 29 22:49:53.390219 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.387732    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd" event={"ID":"44273e03-eb0c-48e0-82ed-9adfc7badd3d","Type":"ContainerDied","Data":"bb7959c54cc9c8a0b46975390a69787da7d8237ed40acb14a3f238c01ac27d39"}
Oct 29 22:49:53.390219 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.387877    4559 scope.go:122] "RemoveContainer" containerID="305fff2921d2a8374ea20fbb65a2458789e31ffc891310c3c058f9df56884ab7"
Oct 29 22:49:53.398795 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.398567    4559 generic.go:356] "Generic (PLEG): container finished" podID="a4109193-5cc4-4dcb-9f8d-fbbe5e359010" containerID="d69054a2db02d913431abd772f1ed9f865efd4bc1a2b8659adb8f86ec6625d06" exitCode=0
Oct 29 22:49:53.398795 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.398631    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"
Oct 29 22:49:53.398795 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.398673    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v" event={"ID":"a4109193-5cc4-4dcb-9f8d-fbbe5e359010","Type":"ContainerDied","Data":"d69054a2db02d913431abd772f1ed9f865efd4bc1a2b8659adb8f86ec6625d06"}
Oct 29 22:49:53.398795 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.398722    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v" event={"ID":"a4109193-5cc4-4dcb-9f8d-fbbe5e359010","Type":"ContainerDied","Data":"9cb99e3fc23dcda23c473a1a81fb306078ff88ab5fe5cc07ff5a2ff7043b40a8"}
Oct 29 22:49:53.629897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.612896    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"]
Oct 29 22:49:53.709468 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.709432    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a3ef0-8648759887-mhb5v"]
Oct 29 22:49:53.747209 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.745000    4559 scope.go:122] "RemoveContainer" containerID="d69054a2db02d913431abd772f1ed9f865efd4bc1a2b8659adb8f86ec6625d06"
Oct 29 22:49:53.771568 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.771533    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:49:53.775234 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.775209    4559 scope.go:122] "RemoveContainer" containerID="d69054a2db02d913431abd772f1ed9f865efd4bc1a2b8659adb8f86ec6625d06"
Oct 29 22:49:53.775637 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:53.775607    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69054a2db02d913431abd772f1ed9f865efd4bc1a2b8659adb8f86ec6625d06\": container with ID starting with d69054a2db02d913431abd772f1ed9f865efd4bc1a2b8659adb8f86ec6625d06 not found: ID does not exist" containerID="d69054a2db02d913431abd772f1ed9f865efd4bc1a2b8659adb8f86ec6625d06"
Oct 29 22:49:53.775731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.775660    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69054a2db02d913431abd772f1ed9f865efd4bc1a2b8659adb8f86ec6625d06"} err="failed to get container status \"d69054a2db02d913431abd772f1ed9f865efd4bc1a2b8659adb8f86ec6625d06\": rpc error: code = NotFound desc = could not find container \"d69054a2db02d913431abd772f1ed9f865efd4bc1a2b8659adb8f86ec6625d06\": container with ID starting with d69054a2db02d913431abd772f1ed9f865efd4bc1a2b8659adb8f86ec6625d06 not found: ID does not exist"
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.834496    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-podinfo\") pod \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") "
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.834554    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-workload-socket\") pod \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") "
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.834667    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-credential-socket\") pod \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") "
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.834721    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-token\" (UniqueName: \"kubernetes.io/projected/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-token\") pod \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") "
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.834755    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-data\") pod \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") "
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.834776    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-workload-socket" pod "44273e03-eb0c-48e0-82ed-9adfc7badd3d" (UID: "44273e03-eb0c-48e0-82ed-9adfc7badd3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.834798    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-workload-certs\") pod \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") "
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.834906    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44273e03-eb0c-48e0-82ed-9adfc7badd3d-kube-api-access-5n2vr\" (UniqueName: \"kubernetes.io/projected/44273e03-eb0c-48e0-82ed-9adfc7badd3d-kube-api-access-5n2vr\") pod \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") "
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.835002    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-envoy\") pod \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") "
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.835033    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istiod-ca-cert\") pod \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\" (UID: \"44273e03-eb0c-48e0-82ed-9adfc7badd3d\") "
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.835074    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-data" pod "44273e03-eb0c-48e0-82ed-9adfc7badd3d" (UID: "44273e03-eb0c-48e0-82ed-9adfc7badd3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.835170    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-workload-certs" pod "44273e03-eb0c-48e0-82ed-9adfc7badd3d" (UID: "44273e03-eb0c-48e0-82ed-9adfc7badd3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.835852    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istiod-ca-cert" pod "44273e03-eb0c-48e0-82ed-9adfc7badd3d" (UID: "44273e03-eb0c-48e0-82ed-9adfc7badd3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.836031    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.836054    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.836068    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.836081    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.836088    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-credential-socket" pod "44273e03-eb0c-48e0-82ed-9adfc7badd3d" (UID: "44273e03-eb0c-48e0-82ed-9adfc7badd3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.837359    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-envoy" pod "44273e03-eb0c-48e0-82ed-9adfc7badd3d" (UID: "44273e03-eb0c-48e0-82ed-9adfc7badd3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.837490    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-podinfo" pod "44273e03-eb0c-48e0-82ed-9adfc7badd3d" (UID: "44273e03-eb0c-48e0-82ed-9adfc7badd3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.838013    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44273e03-eb0c-48e0-82ed-9adfc7badd3d-kube-api-access-5n2vr" pod "44273e03-eb0c-48e0-82ed-9adfc7badd3d" (UID: "44273e03-eb0c-48e0-82ed-9adfc7badd3d"). InnerVolumeSpecName "kube-api-access-5n2vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:53.845848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.838346    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-token" pod "44273e03-eb0c-48e0-82ed-9adfc7badd3d" (UID: "44273e03-eb0c-48e0-82ed-9adfc7badd3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:53.937846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.937778    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n2vr\" (UniqueName: \"kubernetes.io/projected/44273e03-eb0c-48e0-82ed-9adfc7badd3d-kube-api-access-5n2vr\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.937846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.937845    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.938118 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.937857    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.938118 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.937866    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44273e03-eb0c-48e0-82ed-9adfc7badd3d-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.938118 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.937874    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44273e03-eb0c-48e0-82ed-9adfc7badd3d-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.995615 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.995551    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"]
Oct 29 22:49:53.995938 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.995889    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp" podUID="33612869-0569-420d-959a-289d3dcbb1db" containerName="istio-proxy" containerID="cri-o://1791557a62b92bbae7bc643c3020884e731db344d85773f7d02a1c696424c623" gracePeriod=30
Oct 29 22:49:53.999712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.999687    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"]
Oct 29 22:49:53.999962 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:53.999934    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr" podUID="598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" containerName="istio-proxy" containerID="cri-o://f741cec91e60c6fa882613f53b0a70507df8efbe074b62b2f432f35cf8b1b3d0" gracePeriod=30
Oct 29 22:49:54.409582 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:54.409548    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"
Oct 29 22:49:54.410117 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:54.409548    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd" event={"ID":"44273e03-eb0c-48e0-82ed-9adfc7badd3d","Type":"ContainerDied","Data":"8b1d265e4ea903fab4037ad034d62bcc83f2270d093d86f45dc20b81b93e2eaf"}
Oct 29 22:49:54.410117 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:54.409715    4559 scope.go:122] "RemoveContainer" containerID="bb7959c54cc9c8a0b46975390a69787da7d8237ed40acb14a3f238c01ac27d39"
Oct 29 22:49:54.467317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:54.467286    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"]
Oct 29 22:49:54.479874 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:54.479845    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-k9ckd"]
Oct 29 22:49:55.229980 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.229949    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44273e03-eb0c-48e0-82ed-9adfc7badd3d" path="/var/lib/kubelet/pods/44273e03-eb0c-48e0-82ed-9adfc7badd3d/volumes"
Oct 29 22:49:55.230445 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.230430    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4109193-5cc4-4dcb-9f8d-fbbe5e359010" path="/var/lib/kubelet/pods/a4109193-5cc4-4dcb-9f8d-fbbe5e359010/volumes"
Oct 29 22:49:55.317427 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.317392    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"]
Oct 29 22:49:55.318212 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.318192    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b6cffbad-4429-4e61-b582-5b45fda07b48" containerName="istio-proxy"
Oct 29 22:49:55.318212 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.318213    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6cffbad-4429-4e61-b582-5b45fda07b48" containerName="istio-proxy"
Oct 29 22:49:55.318413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.318254    4559 cpu_manager.go:414] "containerMap: removing container" podUID="44273e03-eb0c-48e0-82ed-9adfc7badd3d" containerName="istio-proxy"
Oct 29 22:49:55.318413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.318262    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44273e03-eb0c-48e0-82ed-9adfc7badd3d" containerName="istio-proxy"
Oct 29 22:49:55.318413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.318284    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a4109193-5cc4-4dcb-9f8d-fbbe5e359010" containerName="payload-pre-processing"
Oct 29 22:49:55.318413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.318292    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4109193-5cc4-4dcb-9f8d-fbbe5e359010" containerName="payload-pre-processing"
Oct 29 22:49:55.318413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.318318    4559 cpu_manager.go:414] "containerMap: removing container" podUID="44273e03-eb0c-48e0-82ed-9adfc7badd3d" containerName="istio-proxy"
Oct 29 22:49:55.318413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.318325    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44273e03-eb0c-48e0-82ed-9adfc7badd3d" containerName="istio-proxy"
Oct 29 22:49:55.318413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.318341    4559 cpu_manager.go:414] "containerMap: removing container" podUID="02cf8b24-105b-4675-a4de-8c8c77e4afa9" containerName="payload-processing"
Oct 29 22:49:55.318413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.318348    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02cf8b24-105b-4675-a4de-8c8c77e4afa9" containerName="payload-processing"
Oct 29 22:49:55.318799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.318502    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4109193-5cc4-4dcb-9f8d-fbbe5e359010" containerName="payload-pre-processing"
Oct 29 22:49:55.318799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.318520    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="44273e03-eb0c-48e0-82ed-9adfc7badd3d" containerName="istio-proxy"
Oct 29 22:49:55.318799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.318532    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6cffbad-4429-4e61-b582-5b45fda07b48" containerName="istio-proxy"
Oct 29 22:49:55.318799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.318541    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="02cf8b24-105b-4675-a4de-8c8c77e4afa9" containerName="payload-processing"
Oct 29 22:49:55.318799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.318555    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6cffbad-4429-4e61-b582-5b45fda07b48" containerName="istio-proxy"
Oct 29 22:49:55.324034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.324014    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.346276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.346225    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-920032-openshift-default-dockercfg-72mrh\""
Oct 29 22:49:55.361777 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.361739    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-workload-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.361973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.361801    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-credential-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.361973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.361921    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istiod-ca-cert\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.362157 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.362134    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-workload-certs\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.362244 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.362178    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-data\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.362334 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.362317    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-podinfo\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.362403 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.362360    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-envoy\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.362403 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.362389    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbg6h\" (UniqueName: \"kubernetes.io/projected/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-kube-api-access-gbg6h\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.362524 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.362453    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-token\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.366103 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.366073    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"]
Oct 29 22:49:55.466838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.463920    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istiod-ca-cert\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.466838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.464013    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-workload-certs\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.466838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.464062    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-data\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.477503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.471065    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istiod-ca-cert\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.477503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.476320    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-data\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.477503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.476602    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-workload-certs\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.477503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.476731    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-podinfo\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.477503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.476777    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-envoy\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.477503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.476828    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbg6h\" (UniqueName: \"kubernetes.io/projected/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-kube-api-access-gbg6h\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.477503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.476928    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-token\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.477503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.477036    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-workload-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.477503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.477080    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-credential-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.477503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.477485    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-credential-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.526361 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.492877    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-workload-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.526361 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.493696    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-podinfo\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.554838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.548157    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-envoy\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.624835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.624003    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-token\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.642836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.633561    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbg6h\" (UniqueName: \"kubernetes.io/projected/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-kube-api-access-gbg6h\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:55.642836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:55.640238    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:56.022868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:56.022834    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"]
Oct 29 22:49:56.023073 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:49:56.023052    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700d065a_5535_41fb_bbc6_ba2fc54cd0a4.slice/crio-a3cf1e8a783a5e9e185f04ba94ae16a6fe1c905a883377520a200f3920fb3d7d WatchSource:0}: Error finding container a3cf1e8a783a5e9e185f04ba94ae16a6fe1c905a883377520a200f3920fb3d7d: Status 404 returned error can't find the container with id a3cf1e8a783a5e9e185f04ba94ae16a6fe1c905a883377520a200f3920fb3d7d
Oct 29 22:49:56.026403 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:56.026365    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:49:56.026488 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:56.026450    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:49:56.026525 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:56.026491    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:49:56.457869 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:56.457779    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp" event={"ID":"700d065a-5535-41fb-bbc6-ba2fc54cd0a4","Type":"ContainerStarted","Data":"3877bfc2b81445c92aea7c65de6a4c43f67e2b5e4febeea6a8e8d4f8dd9531a8"}
Oct 29 22:49:56.458154 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:56.457895    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp" event={"ID":"700d065a-5535-41fb-bbc6-ba2fc54cd0a4","Type":"ContainerStarted","Data":"a3cf1e8a783a5e9e185f04ba94ae16a6fe1c905a883377520a200f3920fb3d7d"}
Oct 29 22:49:56.506472 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:56.506417    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp" podStartSLOduration=1.506400003 podStartE2EDuration="1.506400003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:56.503144362 +0000 UTC m=+2135.414587938" watchObservedRunningTime="2026-10-29 22:49:56.506400003 +0000 UTC m=+2135.417843557"
Oct 29 22:49:56.651006 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:56.650921    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:57.648320 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:57.648285    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:58.475294 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:58.475253    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:58.476503 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:58.476481    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:49:59.074337 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:59.074293    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598c2f2f_ccef_4d28_a82b_b8fb8fe5d170.slice/crio-conmon-f741cec91e60c6fa882613f53b0a70507df8efbe074b62b2f432f35cf8b1b3d0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598c2f2f_ccef_4d28_a82b_b8fb8fe5d170.slice/crio-f741cec91e60c6fa882613f53b0a70507df8efbe074b62b2f432f35cf8b1b3d0.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:49:59.074715 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:59.074353    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598c2f2f_ccef_4d28_a82b_b8fb8fe5d170.slice/crio-f741cec91e60c6fa882613f53b0a70507df8efbe074b62b2f432f35cf8b1b3d0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598c2f2f_ccef_4d28_a82b_b8fb8fe5d170.slice/crio-conmon-f741cec91e60c6fa882613f53b0a70507df8efbe074b62b2f432f35cf8b1b3d0.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:49:59.332094 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.332067    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:49:59.335877 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.335852    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:59.480913 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.480881    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-credential-socket\") pod \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") "
Oct 29 22:49:59.481179 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481158    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-data\") pod \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") "
Oct 29 22:49:59.481276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481188    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-credential-socket" pod "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" (UID: "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.481276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481207    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-workload-certs\") pod \"33612869-0569-420d-959a-289d3dcbb1db\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") "
Oct 29 22:49:59.481387 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481373    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-workload-certs\") pod \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") "
Oct 29 22:49:59.481460 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481445    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-podinfo\") pod \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") "
Oct 29 22:49:59.481585 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481570    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-workload-socket\") pod \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") "
Oct 29 22:49:59.481664 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481612    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-kube-api-access-vhspf\" (UniqueName: \"kubernetes.io/projected/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-kube-api-access-vhspf\") pod \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") "
Oct 29 22:49:59.481664 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481656    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33612869-0569-420d-959a-289d3dcbb1db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33612869-0569-420d-959a-289d3dcbb1db-istiod-ca-cert\") pod \"33612869-0569-420d-959a-289d3dcbb1db\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") "
Oct 29 22:49:59.481764 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481722    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istiod-ca-cert\") pod \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") "
Oct 29 22:49:59.481764 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481758    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33612869-0569-420d-959a-289d3dcbb1db-istio-token\" (UniqueName: \"kubernetes.io/projected/33612869-0569-420d-959a-289d3dcbb1db-istio-token\") pod \"33612869-0569-420d-959a-289d3dcbb1db\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") "
Oct 29 22:49:59.481937 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481784    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-token\" (UniqueName: \"kubernetes.io/projected/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-token\") pod \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") "
Oct 29 22:49:59.481937 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481842    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-envoy\") pod \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\" (UID: \"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170\") "
Oct 29 22:49:59.481937 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481875    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-istio-data\") pod \"33612869-0569-420d-959a-289d3dcbb1db\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") "
Oct 29 22:49:59.481937 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481915    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-credential-socket\") pod \"33612869-0569-420d-959a-289d3dcbb1db\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") "
Oct 29 22:49:59.482156 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481955    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33612869-0569-420d-959a-289d3dcbb1db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33612869-0569-420d-959a-289d3dcbb1db-istio-podinfo\") pod \"33612869-0569-420d-959a-289d3dcbb1db\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") "
Oct 29 22:49:59.482156 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.482003    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-istio-envoy\") pod \"33612869-0569-420d-959a-289d3dcbb1db\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") "
Oct 29 22:49:59.482156 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.482095    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33612869-0569-420d-959a-289d3dcbb1db-kube-api-access-stfqh\" (UniqueName: \"kubernetes.io/projected/33612869-0569-420d-959a-289d3dcbb1db-kube-api-access-stfqh\") pod \"33612869-0569-420d-959a-289d3dcbb1db\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") "
Oct 29 22:49:59.482156 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.482130    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-workload-socket\") pod \"33612869-0569-420d-959a-289d3dcbb1db\" (UID: \"33612869-0569-420d-959a-289d3dcbb1db\") "
Oct 29 22:49:59.484881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481372    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-data" pod "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" (UID: "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.484881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481541    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-workload-certs" pod "33612869-0569-420d-959a-289d3dcbb1db" (UID: "33612869-0569-420d-959a-289d3dcbb1db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.484881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481740    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-workload-certs" pod "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" (UID: "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.484881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.481950    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-workload-socket" pod "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" (UID: "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.484881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.482141    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istiod-ca-cert" pod "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" (UID: "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:59.484881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.482501    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-credential-socket" pod "33612869-0569-420d-959a-289d3dcbb1db" (UID: "33612869-0569-420d-959a-289d3dcbb1db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.484881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.483016    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-workload-socket" pod "33612869-0569-420d-959a-289d3dcbb1db" (UID: "33612869-0569-420d-959a-289d3dcbb1db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.484881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.483574    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.484881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.483600    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.484881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.483617    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.484881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.483630    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.484881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.483642    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.484881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.483655    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.484881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.483668    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.484881 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.483680    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.486412 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.485962    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33612869-0569-420d-959a-289d3dcbb1db-kube-api-access-stfqh" pod "33612869-0569-420d-959a-289d3dcbb1db" (UID: "33612869-0569-420d-959a-289d3dcbb1db"). InnerVolumeSpecName "kube-api-access-stfqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:59.486412 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.486130    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33612869-0569-420d-959a-289d3dcbb1db-istiod-ca-cert" pod "33612869-0569-420d-959a-289d3dcbb1db" (UID: "33612869-0569-420d-959a-289d3dcbb1db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:59.486412 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.486330    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-istio-data" pod "33612869-0569-420d-959a-289d3dcbb1db" (UID: "33612869-0569-420d-959a-289d3dcbb1db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.486785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.486759    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-podinfo" pod "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" (UID: "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:59.486910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.486858    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-kube-api-access-vhspf" pod "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" (UID: "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170"). InnerVolumeSpecName "kube-api-access-vhspf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:59.487143 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.487122    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-envoy" pod "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" (UID: "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.487903 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.487880    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33612869-0569-420d-959a-289d3dcbb1db-istio-token" pod "33612869-0569-420d-959a-289d3dcbb1db" (UID: "33612869-0569-420d-959a-289d3dcbb1db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:59.488272 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.488250    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-token" pod "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" (UID: "598c2f2f-ccef-4d28-a82b-b8fb8fe5d170"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:59.490546 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.490286    4559 generic.go:356] "Generic (PLEG): container finished" podID="33612869-0569-420d-959a-289d3dcbb1db" containerID="1791557a62b92bbae7bc643c3020884e731db344d85773f7d02a1c696424c623" exitCode=0
Oct 29 22:49:59.490546 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.490351    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"
Oct 29 22:49:59.490546 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.490371    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp" event={"ID":"33612869-0569-420d-959a-289d3dcbb1db","Type":"ContainerDied","Data":"1791557a62b92bbae7bc643c3020884e731db344d85773f7d02a1c696424c623"}
Oct 29 22:49:59.490546 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.490433    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp" event={"ID":"33612869-0569-420d-959a-289d3dcbb1db","Type":"ContainerDied","Data":"6768389651db408d4a9b163ef269c77f60f0994ac2bd2a344f0e16a090966e12"}
Oct 29 22:49:59.490546 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.490465    4559 scope.go:122] "RemoveContainer" containerID="1791557a62b92bbae7bc643c3020884e731db344d85773f7d02a1c696424c623"
Oct 29 22:49:59.493354 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.493118    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-istio-envoy" pod "33612869-0569-420d-959a-289d3dcbb1db" (UID: "33612869-0569-420d-959a-289d3dcbb1db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.493354 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.493174    4559 generic.go:356] "Generic (PLEG): container finished" podID="598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" containerID="f741cec91e60c6fa882613f53b0a70507df8efbe074b62b2f432f35cf8b1b3d0" exitCode=0
Oct 29 22:49:59.493354 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.493225    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"
Oct 29 22:49:59.493354 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.493259    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr" event={"ID":"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170","Type":"ContainerDied","Data":"f741cec91e60c6fa882613f53b0a70507df8efbe074b62b2f432f35cf8b1b3d0"}
Oct 29 22:49:59.493354 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.493297    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr" event={"ID":"598c2f2f-ccef-4d28-a82b-b8fb8fe5d170","Type":"ContainerDied","Data":"4837704cb7faafe577542cf30c03077e56949c2f2e6d0c667f82cc63845738ed"}
Oct 29 22:49:59.496937 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.496909    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33612869-0569-420d-959a-289d3dcbb1db-istio-podinfo" pod "33612869-0569-420d-959a-289d3dcbb1db" (UID: "33612869-0569-420d-959a-289d3dcbb1db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:59.519749 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.519726    4559 scope.go:122] "RemoveContainer" containerID="1791557a62b92bbae7bc643c3020884e731db344d85773f7d02a1c696424c623"
Oct 29 22:49:59.520104 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:59.520083    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1791557a62b92bbae7bc643c3020884e731db344d85773f7d02a1c696424c623\": container with ID starting with 1791557a62b92bbae7bc643c3020884e731db344d85773f7d02a1c696424c623 not found: ID does not exist" containerID="1791557a62b92bbae7bc643c3020884e731db344d85773f7d02a1c696424c623"
Oct 29 22:49:59.520207 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.520115    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1791557a62b92bbae7bc643c3020884e731db344d85773f7d02a1c696424c623"} err="failed to get container status \"1791557a62b92bbae7bc643c3020884e731db344d85773f7d02a1c696424c623\": rpc error: code = NotFound desc = could not find container \"1791557a62b92bbae7bc643c3020884e731db344d85773f7d02a1c696424c623\": container with ID starting with 1791557a62b92bbae7bc643c3020884e731db344d85773f7d02a1c696424c623 not found: ID does not exist"
Oct 29 22:49:59.520207 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.520142    4559 scope.go:122] "RemoveContainer" containerID="f741cec91e60c6fa882613f53b0a70507df8efbe074b62b2f432f35cf8b1b3d0"
Oct 29 22:49:59.531454 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.531421    4559 scope.go:122] "RemoveContainer" containerID="f741cec91e60c6fa882613f53b0a70507df8efbe074b62b2f432f35cf8b1b3d0"
Oct 29 22:49:59.531746 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:49:59.531727    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f741cec91e60c6fa882613f53b0a70507df8efbe074b62b2f432f35cf8b1b3d0\": container with ID starting with f741cec91e60c6fa882613f53b0a70507df8efbe074b62b2f432f35cf8b1b3d0 not found: ID does not exist" containerID="f741cec91e60c6fa882613f53b0a70507df8efbe074b62b2f432f35cf8b1b3d0"
Oct 29 22:49:59.531902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.531764    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f741cec91e60c6fa882613f53b0a70507df8efbe074b62b2f432f35cf8b1b3d0"} err="failed to get container status \"f741cec91e60c6fa882613f53b0a70507df8efbe074b62b2f432f35cf8b1b3d0\": rpc error: code = NotFound desc = could not find container \"f741cec91e60c6fa882613f53b0a70507df8efbe074b62b2f432f35cf8b1b3d0\": container with ID starting with f741cec91e60c6fa882613f53b0a70507df8efbe074b62b2f432f35cf8b1b3d0 not found: ID does not exist"
Oct 29 22:49:59.562424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.562336    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"]
Oct 29 22:49:59.567556 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.567525    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-ml8sr"]
Oct 29 22:49:59.585286 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.585250    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.585286 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.585289    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhspf\" (UniqueName: \"kubernetes.io/projected/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-kube-api-access-vhspf\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.585522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.585306    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33612869-0569-420d-959a-289d3dcbb1db-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.585522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.585321    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33612869-0569-420d-959a-289d3dcbb1db-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.585522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.585334    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.585522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.585347    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.585522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.585360    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.585522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.585373    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33612869-0569-420d-959a-289d3dcbb1db-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.585522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.585386    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33612869-0569-420d-959a-289d3dcbb1db-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.585522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.585400    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stfqh\" (UniqueName: \"kubernetes.io/projected/33612869-0569-420d-959a-289d3dcbb1db-kube-api-access-stfqh\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.842995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.842902    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"]
Oct 29 22:49:59.845842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:49:59.845795    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-wghsp"]
Oct 29 22:50:00.822965 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:00.822930    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"]
Oct 29 22:50:00.823664 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:00.823646    4559 cpu_manager.go:414] "containerMap: removing container" podUID="598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" containerName="istio-proxy"
Oct 29 22:50:00.823664 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:00.823665    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" containerName="istio-proxy"
Oct 29 22:50:00.823752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:00.823688    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b6cffbad-4429-4e61-b582-5b45fda07b48" containerName="istio-proxy"
Oct 29 22:50:00.823752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:00.823696    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6cffbad-4429-4e61-b582-5b45fda07b48" containerName="istio-proxy"
Oct 29 22:50:00.823752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:00.823719    4559 cpu_manager.go:414] "containerMap: removing container" podUID="33612869-0569-420d-959a-289d3dcbb1db" containerName="istio-proxy"
Oct 29 22:50:00.823752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:00.823725    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33612869-0569-420d-959a-289d3dcbb1db" containerName="istio-proxy"
Oct 29 22:50:00.823982 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:00.823863    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="33612869-0569-420d-959a-289d3dcbb1db" containerName="istio-proxy"
Oct 29 22:50:00.823982 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:00.823874    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="44273e03-eb0c-48e0-82ed-9adfc7badd3d" containerName="istio-proxy"
Oct 29 22:50:00.823982 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:00.823882    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" containerName="istio-proxy"
Oct 29 22:50:00.829079 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:00.829058    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"
Oct 29 22:50:00.852098 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:00.848477    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-920032\""
Oct 29 22:50:00.852795 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:00.852773    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"]
Oct 29 22:50:00.891502 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:00.891445    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-920032-dockercfg-ldcvb\""
Oct 29 22:50:01.007838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.007096    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkz8b\" (UniqueName: \"kubernetes.io/projected/290fd117-eaa7-4003-859d-3dd8ad075e05-kube-api-access-lkz8b\") pod \"payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62\" (UID: \"290fd117-eaa7-4003-859d-3dd8ad075e05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"
Oct 29 22:50:01.007838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.007168    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290fd117-eaa7-4003-859d-3dd8ad075e05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62\" (UID: \"290fd117-eaa7-4003-859d-3dd8ad075e05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"
Oct 29 22:50:01.149887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.149716    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"]
Oct 29 22:50:01.157678 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.157639    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"
Oct 29 22:50:01.177972 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.172688    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkz8b\" (UniqueName: \"kubernetes.io/projected/290fd117-eaa7-4003-859d-3dd8ad075e05-kube-api-access-lkz8b\") pod \"payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62\" (UID: \"290fd117-eaa7-4003-859d-3dd8ad075e05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"
Oct 29 22:50:01.177972 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.172760    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290fd117-eaa7-4003-859d-3dd8ad075e05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62\" (UID: \"290fd117-eaa7-4003-859d-3dd8ad075e05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"
Oct 29 22:50:01.181491 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.181451    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290fd117-eaa7-4003-859d-3dd8ad075e05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62\" (UID: \"290fd117-eaa7-4003-859d-3dd8ad075e05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"
Oct 29 22:50:01.194747 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.194716    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkz8b\" (UniqueName: \"kubernetes.io/projected/290fd117-eaa7-4003-859d-3dd8ad075e05-kube-api-access-lkz8b\") pod \"payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62\" (UID: \"290fd117-eaa7-4003-859d-3dd8ad075e05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"
Oct 29 22:50:01.208148 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.207863    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"]
Oct 29 22:50:01.246532 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.246503    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33612869-0569-420d-959a-289d3dcbb1db" path="/var/lib/kubelet/pods/33612869-0569-420d-959a-289d3dcbb1db/volumes"
Oct 29 22:50:01.247234 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.247215    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="598c2f2f-ccef-4d28-a82b-b8fb8fe5d170" path="/var/lib/kubelet/pods/598c2f2f-ccef-4d28-a82b-b8fb8fe5d170/volumes"
Oct 29 22:50:01.294456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.294414    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-262h8\" (UniqueName: \"kubernetes.io/projected/fd73261d-a0be-4df4-99f6-73d2ea1c425a-kube-api-access-262h8\") pod \"payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765\" (UID: \"fd73261d-a0be-4df4-99f6-73d2ea1c425a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"
Oct 29 22:50:01.294656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.294495    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd73261d-a0be-4df4-99f6-73d2ea1c425a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765\" (UID: \"fd73261d-a0be-4df4-99f6-73d2ea1c425a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"
Oct 29 22:50:01.396694 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.396652    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-262h8\" (UniqueName: \"kubernetes.io/projected/fd73261d-a0be-4df4-99f6-73d2ea1c425a-kube-api-access-262h8\") pod \"payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765\" (UID: \"fd73261d-a0be-4df4-99f6-73d2ea1c425a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"
Oct 29 22:50:01.396924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.396750    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd73261d-a0be-4df4-99f6-73d2ea1c425a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765\" (UID: \"fd73261d-a0be-4df4-99f6-73d2ea1c425a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"
Oct 29 22:50:01.398108 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.398039    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd73261d-a0be-4df4-99f6-73d2ea1c425a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765\" (UID: \"fd73261d-a0be-4df4-99f6-73d2ea1c425a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"
Oct 29 22:50:01.418348 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.418268    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-262h8\" (UniqueName: \"kubernetes.io/projected/fd73261d-a0be-4df4-99f6-73d2ea1c425a-kube-api-access-262h8\") pod \"payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765\" (UID: \"fd73261d-a0be-4df4-99f6-73d2ea1c425a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"
Oct 29 22:50:01.445306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.445268    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"
Oct 29 22:50:01.524797 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.524568    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"
Oct 29 22:50:01.763951 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.761211    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"]
Oct 29 22:50:01.763951 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.761517    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6" podUID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerName="istio-proxy" containerID="cri-o://9fac65e28cbef7fcf82d8ff76f94f436a569419a092fdf83b987807148854ddd" gracePeriod=30
Oct 29 22:50:01.834711 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:50:01.834190    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290fd117_eaa7_4003_859d_3dd8ad075e05.slice/crio-49f1ca29624a7e39d75654ecb635c360779ff447843c10178a5201943eb6bd78 WatchSource:0}: Error finding container 49f1ca29624a7e39d75654ecb635c360779ff447843c10178a5201943eb6bd78: Status 404 returned error can't find the container with id 49f1ca29624a7e39d75654ecb635c360779ff447843c10178a5201943eb6bd78
Oct 29 22:50:01.836172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:01.836141    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"]
Oct 29 22:50:02.009937 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:02.009893    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"]
Oct 29 22:50:02.245258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:02.245014    4559 patch_prober.go:29] interesting pod/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:50:02.245258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:02.245094    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6" podUID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:50:02.567638 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:02.567591    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62" event={"ID":"290fd117-eaa7-4003-859d-3dd8ad075e05","Type":"ContainerStarted","Data":"966dcdad4418935fdcb245ba3a4eea708534bd4b09a0e78bf9af2834381f8aa1"}
Oct 29 22:50:02.567886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:02.567661    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62" event={"ID":"290fd117-eaa7-4003-859d-3dd8ad075e05","Type":"ContainerStarted","Data":"49f1ca29624a7e39d75654ecb635c360779ff447843c10178a5201943eb6bd78"}
Oct 29 22:50:02.567886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:02.567689    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"
Oct 29 22:50:02.569654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:02.569624    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765" event={"ID":"fd73261d-a0be-4df4-99f6-73d2ea1c425a","Type":"ContainerStarted","Data":"b059f7beee73e4cdbf97ac042eacdeeb9a69324b82b9965eb6594230bf048b0b"}
Oct 29 22:50:02.569787 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:02.569681    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"
Oct 29 22:50:02.569787 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:02.569701    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765" event={"ID":"fd73261d-a0be-4df4-99f6-73d2ea1c425a","Type":"ContainerStarted","Data":"3c6ea2a7ad01477bcb001210266b9113e034bb0593130a866a1c0e5f869d6b52"}
Oct 29 22:50:02.607163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:02.607107    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62" podStartSLOduration=2.607091829 podStartE2EDuration="2.607091829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:02.606509769 +0000 UTC m=+2141.517953329" watchObservedRunningTime="2026-10-29 22:50:02.607091829 +0000 UTC m=+2141.518535424"
Oct 29 22:50:02.647696 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:02.647636    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765" podStartSLOduration=2.647619875 podStartE2EDuration="2.647619875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:02.646135424 +0000 UTC m=+2141.557578982" watchObservedRunningTime="2026-10-29 22:50:02.647619875 +0000 UTC m=+2141.559063429"
Oct 29 22:50:03.040467 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.040429    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"]
Oct 29 22:50:03.046403 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.046371    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.084542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.071067    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-eb02df-openshift-default-dockercfg-8bq6z\""
Oct 29 22:50:03.090969 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.090594    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"]
Oct 29 22:50:03.113112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.112781    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9022f52-574d-478f-ae49-29732f126feb-istio-podinfo\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.113112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.112922    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-workload-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.113112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.112964    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9022f52-574d-478f-ae49-29732f126feb-istio-token\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.113365 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.113129    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9022f52-574d-478f-ae49-29732f126feb-istiod-ca-cert\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.113365 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.113180    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-istio-envoy\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.113452 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.113429    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-workload-certs\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.113518 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.113499    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-credential-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.113595 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.113570    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-istio-data\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.113637 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.113607    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9wb7\" (UniqueName: \"kubernetes.io/projected/d9022f52-574d-478f-ae49-29732f126feb-kube-api-access-k9wb7\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.220835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.220546    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-workload-certs\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.220835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.220664    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-credential-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.220835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.220709    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-istio-data\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.220835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.220738    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9wb7\" (UniqueName: \"kubernetes.io/projected/d9022f52-574d-478f-ae49-29732f126feb-kube-api-access-k9wb7\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.221239 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.220900    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9022f52-574d-478f-ae49-29732f126feb-istio-podinfo\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.221239 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.220931    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-workload-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.221239 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.220958    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9022f52-574d-478f-ae49-29732f126feb-istio-token\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.221239 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.221041    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9022f52-574d-478f-ae49-29732f126feb-istiod-ca-cert\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.221239 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.221098    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-istio-envoy\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.247274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.247239    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-workload-certs\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.252857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.249140    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-credential-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.252857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.249393    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-istio-data\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.252857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.252528    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9022f52-574d-478f-ae49-29732f126feb-istiod-ca-cert\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.253295 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.253256    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-workload-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.334889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.328605    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9wb7\" (UniqueName: \"kubernetes.io/projected/d9022f52-574d-478f-ae49-29732f126feb-kube-api-access-k9wb7\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.355099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.335664    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9022f52-574d-478f-ae49-29732f126feb-istio-podinfo\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.355099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.350237    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-istio-envoy\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.409743 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.409698    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9022f52-574d-478f-ae49-29732f126feb-istio-token\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.500679 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.500395    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:03.862110 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.862052    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"]
Oct 29 22:50:03.863155 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:50:03.863126    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9022f52_574d_478f_ae49_29732f126feb.slice/crio-97a03931317bd577c56446b73de6177e2fd0e22b657ccfd35f2a1ba87d31dd69 WatchSource:0}: Error finding container 97a03931317bd577c56446b73de6177e2fd0e22b657ccfd35f2a1ba87d31dd69: Status 404 returned error can't find the container with id 97a03931317bd577c56446b73de6177e2fd0e22b657ccfd35f2a1ba87d31dd69
Oct 29 22:50:03.866415 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.866382    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:50:03.866527 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.866473    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:50:03.866527 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:03.866520    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:50:04.590391 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:04.590348    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv" event={"ID":"d9022f52-574d-478f-ae49-29732f126feb","Type":"ContainerStarted","Data":"9b08ddfa21728096ee900f21c3d2244423dc355eb363d052953db2589ea315a2"}
Oct 29 22:50:04.590879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:04.590417    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv" event={"ID":"d9022f52-574d-478f-ae49-29732f126feb","Type":"ContainerStarted","Data":"97a03931317bd577c56446b73de6177e2fd0e22b657ccfd35f2a1ba87d31dd69"}
Oct 29 22:50:04.648974 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:04.648919    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv" podStartSLOduration=1.648902375 podStartE2EDuration="1.648902375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:04.646383152 +0000 UTC m=+2143.557826708" watchObservedRunningTime="2026-10-29 22:50:04.648902375 +0000 UTC m=+2143.560345929"
Oct 29 22:50:05.502777 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:05.502740    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:05.509277 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:05.509250    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:05.608104 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:05.608073    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:05.609137 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:05.609119    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:50:07.287440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.287413    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:50:07.428204 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.428177    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d39dc4-cb6c-4265-bcb5-4356168496fb-kube-api-access-5pcmq\" (UniqueName: \"kubernetes.io/projected/81d39dc4-cb6c-4265-bcb5-4356168496fb-kube-api-access-5pcmq\") pod \"81d39dc4-cb6c-4265-bcb5-4356168496fb\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") "
Oct 29 22:50:07.428415 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.428242    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-workload-socket\") pod \"81d39dc4-cb6c-4265-bcb5-4356168496fb\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") "
Oct 29 22:50:07.428415 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.428264    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-workload-certs\") pod \"81d39dc4-cb6c-4265-bcb5-4356168496fb\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") "
Oct 29 22:50:07.428415 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.428334    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-data\") pod \"81d39dc4-cb6c-4265-bcb5-4356168496fb\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") "
Oct 29 22:50:07.428609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.428469    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-podinfo\") pod \"81d39dc4-cb6c-4265-bcb5-4356168496fb\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") "
Oct 29 22:50:07.428609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.428527    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-envoy\") pod \"81d39dc4-cb6c-4265-bcb5-4356168496fb\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") "
Oct 29 22:50:07.428609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.428565    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-workload-socket" pod "81d39dc4-cb6c-4265-bcb5-4356168496fb" (UID: "81d39dc4-cb6c-4265-bcb5-4356168496fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:07.428609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.428576    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-workload-certs" pod "81d39dc4-cb6c-4265-bcb5-4356168496fb" (UID: "81d39dc4-cb6c-4265-bcb5-4356168496fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:07.428609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.428597    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-credential-socket\") pod \"81d39dc4-cb6c-4265-bcb5-4356168496fb\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") "
Oct 29 22:50:07.428609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.428597    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-data" pod "81d39dc4-cb6c-4265-bcb5-4356168496fb" (UID: "81d39dc4-cb6c-4265-bcb5-4356168496fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:07.428995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.428644    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-token\" (UniqueName: \"kubernetes.io/projected/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-token\") pod \"81d39dc4-cb6c-4265-bcb5-4356168496fb\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") "
Oct 29 22:50:07.428995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.428687    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d39dc4-cb6c-4265-bcb5-4356168496fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d39dc4-cb6c-4265-bcb5-4356168496fb-istiod-ca-cert\") pod \"81d39dc4-cb6c-4265-bcb5-4356168496fb\" (UID: \"81d39dc4-cb6c-4265-bcb5-4356168496fb\") "
Oct 29 22:50:07.428995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.428908    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-credential-socket" pod "81d39dc4-cb6c-4265-bcb5-4356168496fb" (UID: "81d39dc4-cb6c-4265-bcb5-4356168496fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:07.429143 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.429068    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d39dc4-cb6c-4265-bcb5-4356168496fb-istiod-ca-cert" pod "81d39dc4-cb6c-4265-bcb5-4356168496fb" (UID: "81d39dc4-cb6c-4265-bcb5-4356168496fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:50:07.429700 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.429676    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.429864 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.429705    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.429864 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.429720    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d39dc4-cb6c-4265-bcb5-4356168496fb-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.429864 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.429752    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.429864 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.429764    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.430858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.430777    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-envoy" pod "81d39dc4-cb6c-4265-bcb5-4356168496fb" (UID: "81d39dc4-cb6c-4265-bcb5-4356168496fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:07.430858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.430791    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d39dc4-cb6c-4265-bcb5-4356168496fb-kube-api-access-5pcmq" pod "81d39dc4-cb6c-4265-bcb5-4356168496fb" (UID: "81d39dc4-cb6c-4265-bcb5-4356168496fb"). InnerVolumeSpecName "kube-api-access-5pcmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:07.430858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.430791    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-podinfo" pod "81d39dc4-cb6c-4265-bcb5-4356168496fb" (UID: "81d39dc4-cb6c-4265-bcb5-4356168496fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:50:07.431105 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.431082    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-token" pod "81d39dc4-cb6c-4265-bcb5-4356168496fb" (UID: "81d39dc4-cb6c-4265-bcb5-4356168496fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:07.531166 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.531125    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.531166 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.531157    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.531166 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.531166    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d39dc4-cb6c-4265-bcb5-4356168496fb-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.531166 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.531176    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pcmq\" (UniqueName: \"kubernetes.io/projected/81d39dc4-cb6c-4265-bcb5-4356168496fb-kube-api-access-5pcmq\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.675441 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.675406    4559 generic.go:356] "Generic (PLEG): container finished" podID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerID="9fac65e28cbef7fcf82d8ff76f94f436a569419a092fdf83b987807148854ddd" exitCode=0
Oct 29 22:50:07.675622 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.675474    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"
Oct 29 22:50:07.675622 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.675492    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6" event={"ID":"81d39dc4-cb6c-4265-bcb5-4356168496fb","Type":"ContainerDied","Data":"9fac65e28cbef7fcf82d8ff76f94f436a569419a092fdf83b987807148854ddd"}
Oct 29 22:50:07.675622 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.675559    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6" event={"ID":"81d39dc4-cb6c-4265-bcb5-4356168496fb","Type":"ContainerDied","Data":"2ef62e357310c18baa18909f5e4b0ca243c2ccd564f3cf254e34b402a1147daa"}
Oct 29 22:50:07.675622 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.675581    4559 scope.go:122] "RemoveContainer" containerID="9fac65e28cbef7fcf82d8ff76f94f436a569419a092fdf83b987807148854ddd"
Oct 29 22:50:07.692363 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.692343    4559 scope.go:122] "RemoveContainer" containerID="50397d1c8c03c78af8db4d8ddc218aaca5002c73f22c9a41a7dc9665adccd445"
Oct 29 22:50:07.705718 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.705075    4559 scope.go:122] "RemoveContainer" containerID="9fac65e28cbef7fcf82d8ff76f94f436a569419a092fdf83b987807148854ddd"
Oct 29 22:50:07.705718 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:50:07.705668    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fac65e28cbef7fcf82d8ff76f94f436a569419a092fdf83b987807148854ddd\": container with ID starting with 9fac65e28cbef7fcf82d8ff76f94f436a569419a092fdf83b987807148854ddd not found: ID does not exist" containerID="9fac65e28cbef7fcf82d8ff76f94f436a569419a092fdf83b987807148854ddd"
Oct 29 22:50:07.705718 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.705702    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fac65e28cbef7fcf82d8ff76f94f436a569419a092fdf83b987807148854ddd"} err="failed to get container status \"9fac65e28cbef7fcf82d8ff76f94f436a569419a092fdf83b987807148854ddd\": rpc error: code = NotFound desc = could not find container \"9fac65e28cbef7fcf82d8ff76f94f436a569419a092fdf83b987807148854ddd\": container with ID starting with 9fac65e28cbef7fcf82d8ff76f94f436a569419a092fdf83b987807148854ddd not found: ID does not exist"
Oct 29 22:50:07.705929 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.705725    4559 scope.go:122] "RemoveContainer" containerID="50397d1c8c03c78af8db4d8ddc218aaca5002c73f22c9a41a7dc9665adccd445"
Oct 29 22:50:07.706112 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:50:07.706092    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50397d1c8c03c78af8db4d8ddc218aaca5002c73f22c9a41a7dc9665adccd445\": container with ID starting with 50397d1c8c03c78af8db4d8ddc218aaca5002c73f22c9a41a7dc9665adccd445 not found: ID does not exist" containerID="50397d1c8c03c78af8db4d8ddc218aaca5002c73f22c9a41a7dc9665adccd445"
Oct 29 22:50:07.706206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.706116    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50397d1c8c03c78af8db4d8ddc218aaca5002c73f22c9a41a7dc9665adccd445"} err="failed to get container status \"50397d1c8c03c78af8db4d8ddc218aaca5002c73f22c9a41a7dc9665adccd445\": rpc error: code = NotFound desc = could not find container \"50397d1c8c03c78af8db4d8ddc218aaca5002c73f22c9a41a7dc9665adccd445\": container with ID starting with 50397d1c8c03c78af8db4d8ddc218aaca5002c73f22c9a41a7dc9665adccd445 not found: ID does not exist"
Oct 29 22:50:07.730953 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.730919    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"]
Oct 29 22:50:07.743263 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.743226    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-fwcr6"]
Oct 29 22:50:07.775417 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:07.775373    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:50:09.229251 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.229216    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d39dc4-cb6c-4265-bcb5-4356168496fb" path="/var/lib/kubelet/pods/81d39dc4-cb6c-4265-bcb5-4356168496fb/volumes"
Oct 29 22:50:09.375221 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.375179    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"]
Oct 29 22:50:09.405130 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.405089    4559 cpu_manager.go:414] "containerMap: removing container" podUID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerName="istio-proxy"
Oct 29 22:50:09.405369 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.405351    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerName="istio-proxy"
Oct 29 22:50:09.405500 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.405489    4559 cpu_manager.go:414] "containerMap: removing container" podUID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerName="istio-proxy"
Oct 29 22:50:09.405572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.405563    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerName="istio-proxy"
Oct 29 22:50:09.405850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.405833    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerName="istio-proxy"
Oct 29 22:50:09.434792 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.434754    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"]
Oct 29 22:50:09.435166 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.435011    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"
Oct 29 22:50:09.454780 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.454740    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"]
Oct 29 22:50:09.483893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.456387    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d39dc4-cb6c-4265-bcb5-4356168496fb" containerName="istio-proxy"
Oct 29 22:50:09.490832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.462935    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-eb02df\""
Oct 29 22:50:09.619424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.619381    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655d0083-da33-47ed-bb81-a3433a69da00-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx\" (UID: \"655d0083-da33-47ed-bb81-a3433a69da00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"
Oct 29 22:50:09.620144 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.620109    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wt7v\" (UniqueName: \"kubernetes.io/projected/655d0083-da33-47ed-bb81-a3433a69da00-kube-api-access-9wt7v\") pod \"payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx\" (UID: \"655d0083-da33-47ed-bb81-a3433a69da00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"
Oct 29 22:50:09.797892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.790541    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655d0083-da33-47ed-bb81-a3433a69da00-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx\" (UID: \"655d0083-da33-47ed-bb81-a3433a69da00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"
Oct 29 22:50:09.797892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.790643    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wt7v\" (UniqueName: \"kubernetes.io/projected/655d0083-da33-47ed-bb81-a3433a69da00-kube-api-access-9wt7v\") pod \"payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx\" (UID: \"655d0083-da33-47ed-bb81-a3433a69da00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"
Oct 29 22:50:09.797892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.792540    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655d0083-da33-47ed-bb81-a3433a69da00-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx\" (UID: \"655d0083-da33-47ed-bb81-a3433a69da00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"
Oct 29 22:50:09.934843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.926628    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"
Oct 29 22:50:09.966835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.964906    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"]
Oct 29 22:50:10.000838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:09.999340    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-eb02df-dockercfg-cj9m5\""
Oct 29 22:50:10.022902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:10.022871    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wt7v\" (UniqueName: \"kubernetes.io/projected/655d0083-da33-47ed-bb81-a3433a69da00-kube-api-access-9wt7v\") pod \"payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx\" (UID: \"655d0083-da33-47ed-bb81-a3433a69da00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"
Oct 29 22:50:10.074601 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:10.074156    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5a59ee-c4bb-4457-8db1-9d098bd4147e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn\" (UID: \"ec5a59ee-c4bb-4457-8db1-9d098bd4147e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"
Oct 29 22:50:10.074601 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:10.074268    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8ppz\" (UniqueName: \"kubernetes.io/projected/ec5a59ee-c4bb-4457-8db1-9d098bd4147e-kube-api-access-h8ppz\") pod \"payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn\" (UID: \"ec5a59ee-c4bb-4457-8db1-9d098bd4147e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"
Oct 29 22:50:10.151042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:10.151002    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"
Oct 29 22:50:10.176700 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:10.176657    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5a59ee-c4bb-4457-8db1-9d098bd4147e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn\" (UID: \"ec5a59ee-c4bb-4457-8db1-9d098bd4147e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"
Oct 29 22:50:10.176885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:10.176763    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8ppz\" (UniqueName: \"kubernetes.io/projected/ec5a59ee-c4bb-4457-8db1-9d098bd4147e-kube-api-access-h8ppz\") pod \"payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn\" (UID: \"ec5a59ee-c4bb-4457-8db1-9d098bd4147e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"
Oct 29 22:50:10.178226 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:10.178111    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5a59ee-c4bb-4457-8db1-9d098bd4147e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn\" (UID: \"ec5a59ee-c4bb-4457-8db1-9d098bd4147e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"
Oct 29 22:50:10.198631 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:10.198566    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8ppz\" (UniqueName: \"kubernetes.io/projected/ec5a59ee-c4bb-4457-8db1-9d098bd4147e-kube-api-access-h8ppz\") pod \"payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn\" (UID: \"ec5a59ee-c4bb-4457-8db1-9d098bd4147e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"
Oct 29 22:50:10.375266 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:10.375152    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"
Oct 29 22:50:10.468523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:10.468480    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"]
Oct 29 22:50:10.474865 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:50:10.474835    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655d0083_da33_47ed_bb81_a3433a69da00.slice/crio-a8199cdf70821b3cb2cf9f298899ef52b0ccc8c44239c512301d24323e8b2c8e WatchSource:0}: Error finding container a8199cdf70821b3cb2cf9f298899ef52b0ccc8c44239c512301d24323e8b2c8e: Status 404 returned error can't find the container with id a8199cdf70821b3cb2cf9f298899ef52b0ccc8c44239c512301d24323e8b2c8e
Oct 29 22:50:10.690619 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:10.690587    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"]
Oct 29 22:50:10.691987 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:50:10.691958    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5a59ee_c4bb_4457_8db1_9d098bd4147e.slice/crio-a98cf20985eef3cfeee201ffe17034301006822be51be822baf867fd25a00041 WatchSource:0}: Error finding container a98cf20985eef3cfeee201ffe17034301006822be51be822baf867fd25a00041: Status 404 returned error can't find the container with id a98cf20985eef3cfeee201ffe17034301006822be51be822baf867fd25a00041
Oct 29 22:50:10.753659 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:10.753620    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx" event={"ID":"655d0083-da33-47ed-bb81-a3433a69da00","Type":"ContainerStarted","Data":"2d4c5e1e7aad9fcb8afb2558b79b0fbb4fc09413189ba94f1f00493c36f85e59"}
Oct 29 22:50:10.753846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:10.753680    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx" event={"ID":"655d0083-da33-47ed-bb81-a3433a69da00","Type":"ContainerStarted","Data":"a8199cdf70821b3cb2cf9f298899ef52b0ccc8c44239c512301d24323e8b2c8e"}
Oct 29 22:50:10.754587 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:10.754567    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"
Oct 29 22:50:10.756271 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:10.756244    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn" event={"ID":"ec5a59ee-c4bb-4457-8db1-9d098bd4147e","Type":"ContainerStarted","Data":"a98cf20985eef3cfeee201ffe17034301006822be51be822baf867fd25a00041"}
Oct 29 22:50:10.795373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:10.795318    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx" podStartSLOduration=1.795302817 podStartE2EDuration="1.795302817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:10.793082199 +0000 UTC m=+2149.704525754" watchObservedRunningTime="2026-10-29 22:50:10.795302817 +0000 UTC m=+2149.706746371"
Oct 29 22:50:11.767820 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:11.767760    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn" event={"ID":"ec5a59ee-c4bb-4457-8db1-9d098bd4147e","Type":"ContainerStarted","Data":"c26327bd9adfcced2c9000b49138ad53851abebd5f3e711a1172975dea493efe"}
Oct 29 22:50:11.768354 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:11.767866    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"
Oct 29 22:50:11.808357 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:11.808297    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn" podStartSLOduration=2.808282531 podStartE2EDuration="2.808282531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:11.806344756 +0000 UTC m=+2150.717788327" watchObservedRunningTime="2026-10-29 22:50:11.808282531 +0000 UTC m=+2150.719726143"
Oct 29 22:50:13.584450 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:13.584410    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"
Oct 29 22:50:13.584968 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:13.584540    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"
Oct 29 22:50:13.702911 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:50:13.702867    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:50:22.780789 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:22.780754    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"
Oct 29 22:50:22.781208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:22.780850    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"
Oct 29 22:50:30.095763 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.095722    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"]
Oct 29 22:50:30.102596 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.102570    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.117801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.117774    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-9aafe6-openshift-default-dockercfg-6g2nq\""
Oct 29 22:50:30.135495 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.135456    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"]
Oct 29 22:50:30.166453 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.166407    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-credential-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.166802 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.166772    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-envoy\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.167111 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.167088    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-data\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.167279 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.167261    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-workload-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.167414 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.167398    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-podinfo\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.167578 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.167561    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-token\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.167729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.167711    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tth5p\" (UniqueName: \"kubernetes.io/projected/e574d0f5-9916-4d71-9d2c-8743336447b3-kube-api-access-tth5p\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.167957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.167935    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e574d0f5-9916-4d71-9d2c-8743336447b3-istiod-ca-cert\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.168184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.168153    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-workload-certs\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.284889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.284846    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e574d0f5-9916-4d71-9d2c-8743336447b3-istiod-ca-cert\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.285102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.284923    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-workload-certs\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.285102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.284978    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-credential-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.285102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.285031    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-envoy\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.285273 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.285119    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-data\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.285273 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.285171    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-workload-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.285449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.285281    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-podinfo\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.285449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.285376    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-token\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.285449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.285418    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tth5p\" (UniqueName: \"kubernetes.io/projected/e574d0f5-9916-4d71-9d2c-8743336447b3-kube-api-access-tth5p\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.285449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.285426    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-workload-certs\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.286067 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.285467    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-credential-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.286067 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.285523    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-workload-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.286067 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.286046    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-data\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.286624 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.286598    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e574d0f5-9916-4d71-9d2c-8743336447b3-istiod-ca-cert\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.288292 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.288266    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-podinfo\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.288682 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.288657    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-envoy\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.305792 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.305753    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-token\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.305989 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.305880    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tth5p\" (UniqueName: \"kubernetes.io/projected/e574d0f5-9916-4d71-9d2c-8743336447b3-kube-api-access-tth5p\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.417329 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.417291    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:30.815707 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:50:30.815671    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode574d0f5_9916_4d71_9d2c_8743336447b3.slice/crio-87bc2e70b7315e862438f8ae835b3fcf077b1bba3565720d65ab475f980640a2 WatchSource:0}: Error finding container 87bc2e70b7315e862438f8ae835b3fcf077b1bba3565720d65ab475f980640a2: Status 404 returned error can't find the container with id 87bc2e70b7315e862438f8ae835b3fcf077b1bba3565720d65ab475f980640a2
Oct 29 22:50:30.815849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.815708    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"]
Oct 29 22:50:30.818434 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.818404    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:50:30.818515 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.818469    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:50:30.818515 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.818501    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:50:30.972410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.972366    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc" event={"ID":"e574d0f5-9916-4d71-9d2c-8743336447b3","Type":"ContainerStarted","Data":"a8a480dbd7be7173ae310afba23046d23b49bb9340506603150ffc180ba42c48"}
Oct 29 22:50:30.972722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:30.972673    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc" event={"ID":"e574d0f5-9916-4d71-9d2c-8743336447b3","Type":"ContainerStarted","Data":"87bc2e70b7315e862438f8ae835b3fcf077b1bba3565720d65ab475f980640a2"}
Oct 29 22:50:31.015324 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:31.015253    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc" podStartSLOduration=1.015233464 podStartE2EDuration="1.015233464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:31.014856961 +0000 UTC m=+2169.926300517" watchObservedRunningTime="2026-10-29 22:50:31.015233464 +0000 UTC m=+2169.926677019"
Oct 29 22:50:31.422884 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:31.419900    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:31.427843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:31.423940    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 22:50:31.427843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:31.424000    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc" podUID="e574d0f5-9916-4d71-9d2c-8743336447b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 22:50:32.418031 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:32.418000    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 22:50:32.418306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:32.418060    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc" podUID="e574d0f5-9916-4d71-9d2c-8743336447b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 22:50:33.417975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:33.417936    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 22:50:33.418401 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:33.417999    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc" podUID="e574d0f5-9916-4d71-9d2c-8743336447b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 22:50:34.418389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:34.418355    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 22:50:34.418852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:34.418409    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc" podUID="e574d0f5-9916-4d71-9d2c-8743336447b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 22:50:35.418533 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:35.418488    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 22:50:35.419079 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:35.418552    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc" podUID="e574d0f5-9916-4d71-9d2c-8743336447b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 22:50:36.418434 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:36.418385    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 22:50:36.418664 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:36.418455    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc" podUID="e574d0f5-9916-4d71-9d2c-8743336447b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 22:50:37.417927 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:37.417889    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 22:50:37.418145 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:37.417957    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc" podUID="e574d0f5-9916-4d71-9d2c-8743336447b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 22:50:38.423205 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:38.423172    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:38.423698 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:38.423659    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:38.424718 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:38.424692    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 22:50:39.844425 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:39.844387    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"]
Oct 29 22:50:39.844981 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:39.844672    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj" podUID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" containerName="istio-proxy" containerID="cri-o://215a4eb15881c6d4dc5f3bb542ccb65fd627872244832d3b55193a74b230b323" gracePeriod=30
Oct 29 22:50:42.181852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:42.181786    4559 generic.go:356] "Generic (PLEG): container finished" podID="b905ef8e-61e8-4693-ae47-1b3940e87b0d" containerID="6345299c7471ecfc26af9d2495cc7bcddb76905dcdefb020bb29f8558b74e683" exitCode=0
Oct 29 22:50:42.182430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:42.181911    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84" event={"ID":"b905ef8e-61e8-4693-ae47-1b3940e87b0d","Type":"ContainerDied","Data":"6345299c7471ecfc26af9d2495cc7bcddb76905dcdefb020bb29f8558b74e683"}
Oct 29 22:50:42.182430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:42.182334    4559 scope.go:122] "RemoveContainer" containerID="6345299c7471ecfc26af9d2495cc7bcddb76905dcdefb020bb29f8558b74e683"
Oct 29 22:50:42.184204 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:42.184171    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:50:42.184323 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:42.184260    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:50:42.184323 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:42.184306    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:50:43.193571 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:43.193535    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84" event={"ID":"b905ef8e-61e8-4693-ae47-1b3940e87b0d","Type":"ContainerStarted","Data":"4c77f80dafa1521bffc45f05ccd21021095594b8cb7e09b51fb036430c2a7822"}
Oct 29 22:50:43.696420 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:50:43.696360    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:50:44.016974 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:44.016875    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:50:44.018456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:44.018425    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 29 22:50:44.018612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:44.018487    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84" podUID="b905ef8e-61e8-4693-ae47-1b3940e87b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 29 22:50:44.970536 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:44.970445    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:50:45.021940 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.021903    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 29 22:50:45.022288 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.022257    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84" podUID="b905ef8e-61e8-4693-ae47-1b3940e87b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 29 22:50:45.211691 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.211652    4559 generic.go:356] "Generic (PLEG): container finished" podID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" containerID="215a4eb15881c6d4dc5f3bb542ccb65fd627872244832d3b55193a74b230b323" exitCode=0
Oct 29 22:50:45.211986 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.211713    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj" event={"ID":"10e5d770-5e34-4c1c-a5e9-3b62d070b61e","Type":"ContainerDied","Data":"215a4eb15881c6d4dc5f3bb542ccb65fd627872244832d3b55193a74b230b323"}
Oct 29 22:50:45.211986 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.211851    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj" event={"ID":"10e5d770-5e34-4c1c-a5e9-3b62d070b61e","Type":"ContainerDied","Data":"39af70143e96e8e9ccd7f627ff95b99320dac7ee0c9bdc04435e98899c7b95d0"}
Oct 29 22:50:45.211986 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.211885    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39af70143e96e8e9ccd7f627ff95b99320dac7ee0c9bdc04435e98899c7b95d0"
Oct 29 22:50:45.211986 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.211905    4559 scope.go:122] "RemoveContainer" containerID="d9cfdee6327770ca5f7ec51add57b9ae09aef756952c8d72cb4731d40fd5a9c7"
Oct 29 22:50:45.277314 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.277282    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:50:45.383109 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.383073    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-kube-api-access-lwmqs\" (UniqueName: \"kubernetes.io/projected/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-kube-api-access-lwmqs\") pod \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") "
Oct 29 22:50:45.383343 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.383131    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-token\" (UniqueName: \"kubernetes.io/projected/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-token\") pod \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") "
Oct 29 22:50:45.383343 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.383196    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istiod-ca-cert\") pod \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") "
Oct 29 22:50:45.383343 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.383248    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-workload-certs\") pod \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") "
Oct 29 22:50:45.383522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.383346    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-envoy\") pod \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") "
Oct 29 22:50:45.383522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.383378    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-credential-socket\") pod \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") "
Oct 29 22:50:45.383522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.383456    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-podinfo\") pod \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") "
Oct 29 22:50:45.383674 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.383565    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-workload-socket\") pod \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") "
Oct 29 22:50:45.383730 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.383682    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-data\") pod \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\" (UID: \"10e5d770-5e34-4c1c-a5e9-3b62d070b61e\") "
Oct 29 22:50:45.384174 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.384117    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-workload-certs" pod "10e5d770-5e34-4c1c-a5e9-3b62d070b61e" (UID: "10e5d770-5e34-4c1c-a5e9-3b62d070b61e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:45.384395 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.384192    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-credential-socket" pod "10e5d770-5e34-4c1c-a5e9-3b62d070b61e" (UID: "10e5d770-5e34-4c1c-a5e9-3b62d070b61e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:45.385144 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.385117    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-workload-socket" pod "10e5d770-5e34-4c1c-a5e9-3b62d070b61e" (UID: "10e5d770-5e34-4c1c-a5e9-3b62d070b61e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:45.386215 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.386139    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:45.386215 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.386192    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:45.386387 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.386228    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istiod-ca-cert" pod "10e5d770-5e34-4c1c-a5e9-3b62d070b61e" (UID: "10e5d770-5e34-4c1c-a5e9-3b62d070b61e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:50:45.386511 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.386484    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-data" pod "10e5d770-5e34-4c1c-a5e9-3b62d070b61e" (UID: "10e5d770-5e34-4c1c-a5e9-3b62d070b61e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:45.387481 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.387452    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-podinfo" pod "10e5d770-5e34-4c1c-a5e9-3b62d070b61e" (UID: "10e5d770-5e34-4c1c-a5e9-3b62d070b61e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:50:45.387980 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.387962    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-kube-api-access-lwmqs" pod "10e5d770-5e34-4c1c-a5e9-3b62d070b61e" (UID: "10e5d770-5e34-4c1c-a5e9-3b62d070b61e"). InnerVolumeSpecName "kube-api-access-lwmqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:45.388796 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.388776    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-token" pod "10e5d770-5e34-4c1c-a5e9-3b62d070b61e" (UID: "10e5d770-5e34-4c1c-a5e9-3b62d070b61e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:45.389025 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.389005    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-envoy" pod "10e5d770-5e34-4c1c-a5e9-3b62d070b61e" (UID: "10e5d770-5e34-4c1c-a5e9-3b62d070b61e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:45.502995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.495433    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwmqs\" (UniqueName: \"kubernetes.io/projected/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-kube-api-access-lwmqs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:45.502995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.495475    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:45.502995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.495491    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:45.502995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.495504    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:45.502995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.495530    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:45.502995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.495542    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:45.502995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:45.495553    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e5d770-5e34-4c1c-a5e9-3b62d070b61e-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:50:46.016798 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:46.016753    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 29 22:50:46.017272 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:46.016842    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84" podUID="b905ef8e-61e8-4693-ae47-1b3940e87b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 29 22:50:46.220953 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:46.220918    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"
Oct 29 22:50:46.284198 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:46.284102    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"]
Oct 29 22:50:46.296984 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:46.296946    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-tczdj"]
Oct 29 22:50:47.017028 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:47.016978    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 29 22:50:47.017515 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:47.017047    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84" podUID="b905ef8e-61e8-4693-ae47-1b3940e87b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 29 22:50:47.232493 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:47.232461    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" path="/var/lib/kubelet/pods/10e5d770-5e34-4c1c-a5e9-3b62d070b61e/volumes"
Oct 29 22:50:48.016941 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:48.016905    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 29 22:50:48.017165 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:48.016972    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84" podUID="b905ef8e-61e8-4693-ae47-1b3940e87b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 29 22:50:49.031361 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:49.031326    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:50:49.032912 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:50:49.032887    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 22:51:00.886945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.886905    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"]
Oct 29 22:51:00.888142 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.888118    4559 cpu_manager.go:414] "containerMap: removing container" podUID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" containerName="istio-proxy"
Oct 29 22:51:00.888199 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.888145    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" containerName="istio-proxy"
Oct 29 22:51:00.888250 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.888211    4559 cpu_manager.go:414] "containerMap: removing container" podUID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" containerName="istio-proxy"
Oct 29 22:51:00.888250 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.888220    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" containerName="istio-proxy"
Oct 29 22:51:00.888411 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.888398    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" containerName="istio-proxy"
Oct 29 22:51:00.888456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.888425    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e5d770-5e34-4c1c-a5e9-3b62d070b61e" containerName="istio-proxy"
Oct 29 22:51:00.894329 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.894302    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:00.913639 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.913604    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"]
Oct 29 22:51:00.920462 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.920430    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-6fc6bd\"/\"openshift-service-ca.crt\""
Oct 29 22:51:00.920665 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.920594    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-6fc6bd\"/\"kube-root-ca.crt\""
Oct 29 22:51:00.926942 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.926911    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-6fc6bd\"/\"default-dockercfg-nzgc4\""
Oct 29 22:51:00.926942 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.926912    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-6fc6bd\"/\"rate-test-model-6fc6bd-kserve-self-signed-certs\""
Oct 29 22:51:00.968437 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.968398    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-model-cache\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:00.968654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.968459    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-dshm\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:00.968654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.968598    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-tmp-dir\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:00.968785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.968721    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzmjz\" (UniqueName: \"kubernetes.io/projected/a5788bfa-4108-4ae8-855c-f56ba146d4f0-kube-api-access-kzmjz\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:00.968868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.968789    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-home\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:00.968940 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:00.968922    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5788bfa-4108-4ae8-855c-f56ba146d4f0-tls-certs\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:01.078832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:01.071701    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-tmp-dir\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:01.078832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:01.071837    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzmjz\" (UniqueName: \"kubernetes.io/projected/a5788bfa-4108-4ae8-855c-f56ba146d4f0-kube-api-access-kzmjz\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:01.078832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:01.071890    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-home\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:01.078832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:01.071962    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5788bfa-4108-4ae8-855c-f56ba146d4f0-tls-certs\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:01.078832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:01.072057    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-model-cache\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:01.078832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:01.072102    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-dshm\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:01.089067 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:01.079751    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-home\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:01.089067 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:01.082207    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-model-cache\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:01.089067 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:01.087740    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5788bfa-4108-4ae8-855c-f56ba146d4f0-tls-certs\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:01.089067 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:01.087829    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-dshm\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:01.089439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:01.089391    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-tmp-dir\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:01.108259 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:01.108218    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzmjz\" (UniqueName: \"kubernetes.io/projected/a5788bfa-4108-4ae8-855c-f56ba146d4f0-kube-api-access-kzmjz\") pod \"rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:01.235481 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:01.235200    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:01.629643 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:01.629609    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"]
Oct 29 22:51:01.629767 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:51:01.629689    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5788bfa_4108_4ae8_855c_f56ba146d4f0.slice/crio-94159f17a996d878e1b3fe02a8a0e110e2fde19afb6281c5fe5bcd0d49149e34 WatchSource:0}: Error finding container 94159f17a996d878e1b3fe02a8a0e110e2fde19afb6281c5fe5bcd0d49149e34: Status 404 returned error can't find the container with id 94159f17a996d878e1b3fe02a8a0e110e2fde19afb6281c5fe5bcd0d49149e34
Oct 29 22:51:02.421880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:02.421839    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf" event={"ID":"a5788bfa-4108-4ae8-855c-f56ba146d4f0","Type":"ContainerStarted","Data":"4a22fade355afaca2ddcca43eb66334c9d339dab6804c64084c65b7535228243"}
Oct 29 22:51:02.421880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:02.421887    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf" event={"ID":"a5788bfa-4108-4ae8-855c-f56ba146d4f0","Type":"ContainerStarted","Data":"94159f17a996d878e1b3fe02a8a0e110e2fde19afb6281c5fe5bcd0d49149e34"}
Oct 29 22:51:02.461959 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:02.461860    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf" podStartSLOduration=2.461846885 podStartE2EDuration="2.461846885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:02.459668526 +0000 UTC m=+2201.371112080" watchObservedRunningTime="2026-10-29 22:51:02.461846885 +0000 UTC m=+2201.373290438"
Oct 29 22:51:03.244717 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:03.244684    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:10.852332 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:10.851222    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:51:11.162092 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.162051    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"]
Oct 29 22:51:11.169665 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.169638    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.194444 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.192649    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"]
Oct 29 22:51:11.199857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.199128    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-02720a-openshift-default-dockercfg-gzwqr\""
Oct 29 22:51:11.244428 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.244388    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:11.259047 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.259005    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:11.275383 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.275344    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09d2003-0997-418a-8fa3-f64c92d28572-istiod-ca-cert\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.275608 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.275402    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09d2003-0997-418a-8fa3-f64c92d28572-istio-token\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.275608 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.275492    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg5sg\" (UniqueName: \"kubernetes.io/projected/e09d2003-0997-418a-8fa3-f64c92d28572-kube-api-access-sg5sg\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.275736 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.275606    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-istio-envoy\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.275799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.275752    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-credential-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.276034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.275799    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-workload-certs\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.276034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.275900    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09d2003-0997-418a-8fa3-f64c92d28572-istio-podinfo\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.276034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.275929    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-workload-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.276034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.275981    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-istio-data\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.385873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.385803    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-istio-data\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.386218 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.386196    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09d2003-0997-418a-8fa3-f64c92d28572-istiod-ca-cert\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.386382 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.386367    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09d2003-0997-418a-8fa3-f64c92d28572-istio-token\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.386569 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.386552    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg5sg\" (UniqueName: \"kubernetes.io/projected/e09d2003-0997-418a-8fa3-f64c92d28572-kube-api-access-sg5sg\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.386740 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.386725    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-istio-envoy\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.387081 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.387062    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-credential-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.387263 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.387247    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-workload-certs\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.387403 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.387386    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09d2003-0997-418a-8fa3-f64c92d28572-istio-podinfo\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.387533 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.387505    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-workload-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.388212 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.388187    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-workload-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.388698 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.388676    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-istio-data\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.389569 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.389543    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09d2003-0997-418a-8fa3-f64c92d28572-istiod-ca-cert\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.400833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.399271    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-credential-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.400833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.400025    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-workload-certs\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.407801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.407771    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09d2003-0997-418a-8fa3-f64c92d28572-istio-podinfo\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.424879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.424361    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09d2003-0997-418a-8fa3-f64c92d28572-istio-token\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.446875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.442169    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-istio-envoy\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.446875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.443570    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg5sg\" (UniqueName: \"kubernetes.io/projected/e09d2003-0997-418a-8fa3-f64c92d28572-kube-api-access-sg5sg\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.502834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.499099    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:11.561058 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.561019    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:51:11.956068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.956030    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"]
Oct 29 22:51:11.976666 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:51:11.976628    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09d2003_0997_418a_8fa3_f64c92d28572.slice/crio-ab9150ff73f98b72bb7561ba185bac703df855158b92d65d5a85839383a8efaf WatchSource:0}: Error finding container ab9150ff73f98b72bb7561ba185bac703df855158b92d65d5a85839383a8efaf: Status 404 returned error can't find the container with id ab9150ff73f98b72bb7561ba185bac703df855158b92d65d5a85839383a8efaf
Oct 29 22:51:11.980044 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.980000    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:51:11.980174 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.980097    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:51:11.980174 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:11.980136    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:51:12.559848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:12.557735    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs" event={"ID":"e09d2003-0997-418a-8fa3-f64c92d28572","Type":"ContainerStarted","Data":"01e3d92ce7e5890894e7e1f66c4380832ad287ef5508844f84b3c986e0ca9472"}
Oct 29 22:51:12.559848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:12.557833    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs" event={"ID":"e09d2003-0997-418a-8fa3-f64c92d28572","Type":"ContainerStarted","Data":"ab9150ff73f98b72bb7561ba185bac703df855158b92d65d5a85839383a8efaf"}
Oct 29 22:51:13.502836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:13.501680    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:13.514857 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:13.514274    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:13.565836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:13.565366    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs" podStartSLOduration=2.565344462 podStartE2EDuration="2.565344462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:12.629168734 +0000 UTC m=+2211.540612324" watchObservedRunningTime="2026-10-29 22:51:13.565344462 +0000 UTC m=+2212.476788022"
Oct 29 22:51:13.602836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:13.590863    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:13.606307 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:13.605974    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:51:13.977902 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:51:13.884559    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:51:17.183875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:17.173875    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"]
Oct 29 22:51:17.223853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:17.221709    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"
Oct 29 22:51:17.256900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:17.256843    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-02720a\""
Oct 29 22:51:17.321334 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:17.311456    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-02720a-dockercfg-b2fr5\""
Oct 29 22:51:17.357836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:17.348742    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2w65\" (UniqueName: \"kubernetes.io/projected/7776b504-8e2a-4138-b03a-60b47a61bd53-kube-api-access-n2w65\") pod \"payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh\" (UID: \"7776b504-8e2a-4138-b03a-60b47a61bd53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"
Oct 29 22:51:17.434683 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:17.434574    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7776b504-8e2a-4138-b03a-60b47a61bd53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh\" (UID: \"7776b504-8e2a-4138-b03a-60b47a61bd53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"
Oct 29 22:51:17.531993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:17.531957    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"]
Oct 29 22:51:17.568840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:17.541825    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2w65\" (UniqueName: \"kubernetes.io/projected/7776b504-8e2a-4138-b03a-60b47a61bd53-kube-api-access-n2w65\") pod \"payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh\" (UID: \"7776b504-8e2a-4138-b03a-60b47a61bd53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"
Oct 29 22:51:17.568840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:17.541924    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7776b504-8e2a-4138-b03a-60b47a61bd53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh\" (UID: \"7776b504-8e2a-4138-b03a-60b47a61bd53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"
Oct 29 22:51:17.568840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:17.546747    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7776b504-8e2a-4138-b03a-60b47a61bd53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh\" (UID: \"7776b504-8e2a-4138-b03a-60b47a61bd53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"
Oct 29 22:51:17.606855 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:17.606678    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2w65\" (UniqueName: \"kubernetes.io/projected/7776b504-8e2a-4138-b03a-60b47a61bd53-kube-api-access-n2w65\") pod \"payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh\" (UID: \"7776b504-8e2a-4138-b03a-60b47a61bd53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"
Oct 29 22:51:17.802440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:17.802161    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"
Oct 29 22:51:18.406834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:18.405924    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"]
Oct 29 22:51:18.703974 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:18.697731    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh" event={"ID":"7776b504-8e2a-4138-b03a-60b47a61bd53","Type":"ContainerStarted","Data":"7e91605dc55e843a62eef0b018cdcf5e0b99d5d67ab76dd59cc486ebd5bd4605"}
Oct 29 22:51:19.771830 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:19.771431    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh" event={"ID":"7776b504-8e2a-4138-b03a-60b47a61bd53","Type":"ContainerStarted","Data":"3e30dfe662325dceb7fedc1bf7e7366c53ab76d86d12dbd752dc966a5245e554"}
Oct 29 22:51:19.784886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:19.773262    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"
Oct 29 22:51:19.822833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:19.817715    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh" podStartSLOduration=2.817695868 podStartE2EDuration="2.817695868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:19.815290533 +0000 UTC m=+2218.726734104" watchObservedRunningTime="2026-10-29 22:51:19.817695868 +0000 UTC m=+2218.729139428"
Oct 29 22:51:21.649110 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:21.649067    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:51:21.649622 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:21.649130    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv" podUID="d9022f52-574d-478f-ae49-29732f126feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:51:23.351825 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:23.351756    4559 scope.go:122] "RemoveContainer" containerID="215a4eb15881c6d4dc5f3bb542ccb65fd627872244832d3b55193a74b230b323"
Oct 29 22:51:23.379252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:23.379223    4559 scope.go:122] "RemoveContainer" containerID="d6333c85feaa7c19946d1a62a9d0e9a1bf35c723db9976e4383fd97f6005462b"
Oct 29 22:51:24.869956 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:24.869848    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp" event={"ID":"700d065a-5535-41fb-bbc6-ba2fc54cd0a4","Type":"ContainerDied","Data":"3877bfc2b81445c92aea7c65de6a4c43f67e2b5e4febeea6a8e8d4f8dd9531a8"}
Oct 29 22:51:24.872391 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:24.872353    4559 scope.go:122] "RemoveContainer" containerID="3877bfc2b81445c92aea7c65de6a4c43f67e2b5e4febeea6a8e8d4f8dd9531a8"
Oct 29 22:51:24.872855 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:24.869802    4559 generic.go:356] "Generic (PLEG): container finished" podID="700d065a-5535-41fb-bbc6-ba2fc54cd0a4" containerID="3877bfc2b81445c92aea7c65de6a4c43f67e2b5e4febeea6a8e8d4f8dd9531a8" exitCode=0
Oct 29 22:51:24.876827 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:24.876745    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:51:24.877076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:24.877047    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:51:24.877245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:24.877224    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:51:24.884113 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:24.884087    4559 generic.go:356] "Generic (PLEG): container finished" podID="d9022f52-574d-478f-ae49-29732f126feb" containerID="9b08ddfa21728096ee900f21c3d2244423dc355eb363d052953db2589ea315a2" exitCode=0
Oct 29 22:51:24.884450 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:24.884313    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv" event={"ID":"d9022f52-574d-478f-ae49-29732f126feb","Type":"ContainerDied","Data":"9b08ddfa21728096ee900f21c3d2244423dc355eb363d052953db2589ea315a2"}
Oct 29 22:51:24.885245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:24.885224    4559 scope.go:122] "RemoveContainer" containerID="9b08ddfa21728096ee900f21c3d2244423dc355eb363d052953db2589ea315a2"
Oct 29 22:51:24.892839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:24.892455    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:51:24.892839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:24.892597    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:51:24.892839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:24.892673    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:51:25.905178 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:25.905143    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp" event={"ID":"700d065a-5535-41fb-bbc6-ba2fc54cd0a4","Type":"ContainerStarted","Data":"8abd15896824d6311f1fd8283b3f3979b899050a7a6404509f65749d4dec5d4f"}
Oct 29 22:51:25.907412 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:25.907383    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv" event={"ID":"d9022f52-574d-478f-ae49-29732f126feb","Type":"ContainerStarted","Data":"9109a8eaded4ef240af04cd37dc62d3eb19ebd6c8978bc90290addbdcb385071"}
Oct 29 22:51:26.502887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:26.502851    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:51:26.503237 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:26.503206    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 29 22:51:26.503314 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:26.503278    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv" podUID="d9022f52-574d-478f-ae49-29732f126feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 29 22:51:26.653841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:26.640973    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:51:26.676838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:26.663989    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 29 22:51:26.676838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:26.664090    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp" podUID="700d065a-5535-41fb-bbc6-ba2fc54cd0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 29 22:51:27.502755 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:27.502713    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 29 22:51:27.503197 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:27.502776    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv" podUID="d9022f52-574d-478f-ae49-29732f126feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 29 22:51:27.644336 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:27.644295    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 29 22:51:27.644642 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:27.644613    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp" podUID="700d065a-5535-41fb-bbc6-ba2fc54cd0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 29 22:51:28.475269 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:28.475232    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:51:28.503408 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:28.503373    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 29 22:51:28.503877 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:28.503441    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv" podUID="d9022f52-574d-478f-ae49-29732f126feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 29 22:51:28.646387 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:28.646347    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 29 22:51:28.646681 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:28.646652    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp" podUID="700d065a-5535-41fb-bbc6-ba2fc54cd0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 29 22:51:29.502840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:29.502379    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 29 22:51:29.502840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:29.502453    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv" podUID="d9022f52-574d-478f-ae49-29732f126feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 29 22:51:29.641351 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:29.641312    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 29 22:51:29.642032 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:29.641990    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp" podUID="700d065a-5535-41fb-bbc6-ba2fc54cd0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 29 22:51:30.103925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:30.103879    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"]
Oct 29 22:51:30.109102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:30.109068    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"]
Oct 29 22:51:30.109287 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:30.109156    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"
Oct 29 22:51:30.125782 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:30.125755    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-3ee017\""
Oct 29 22:51:30.228184 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:30.228147    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlx4d\" (UniqueName: \"kubernetes.io/projected/d190385b-94ea-4b39-9318-3a33e14e8286-kube-api-access-jlx4d\") pod \"payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq\" (UID: \"d190385b-94ea-4b39-9318-3a33e14e8286\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"
Oct 29 22:51:30.228667 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:30.228635    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d190385b-94ea-4b39-9318-3a33e14e8286-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq\" (UID: \"d190385b-94ea-4b39-9318-3a33e14e8286\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"
Oct 29 22:51:30.341844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:30.334106    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d190385b-94ea-4b39-9318-3a33e14e8286-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq\" (UID: \"d190385b-94ea-4b39-9318-3a33e14e8286\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"
Oct 29 22:51:30.395000 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:30.355111    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d190385b-94ea-4b39-9318-3a33e14e8286-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq\" (UID: \"d190385b-94ea-4b39-9318-3a33e14e8286\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"
Oct 29 22:51:30.441917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:30.414730    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlx4d\" (UniqueName: \"kubernetes.io/projected/d190385b-94ea-4b39-9318-3a33e14e8286-kube-api-access-jlx4d\") pod \"payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq\" (UID: \"d190385b-94ea-4b39-9318-3a33e14e8286\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"
Oct 29 22:51:30.509430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:30.509297    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 29 22:51:30.509430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:30.509360    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv" podUID="d9022f52-574d-478f-ae49-29732f126feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 29 22:51:30.576011 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:30.529580    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlx4d\" (UniqueName: \"kubernetes.io/projected/d190385b-94ea-4b39-9318-3a33e14e8286-kube-api-access-jlx4d\") pod \"payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq\" (UID: \"d190385b-94ea-4b39-9318-3a33e14e8286\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"
Oct 29 22:51:30.664077 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:30.663972    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 29 22:51:30.689009 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:30.688928    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp" podUID="700d065a-5535-41fb-bbc6-ba2fc54cd0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 29 22:51:30.727194 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:30.727159    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"
Oct 29 22:51:31.500686 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:31.500639    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"]
Oct 29 22:51:31.511020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:31.502580    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 29 22:51:31.511020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:31.502646    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv" podUID="d9022f52-574d-478f-ae49-29732f126feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 29 22:51:31.669836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:31.669616    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 29 22:51:31.669836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:31.669702    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp" podUID="700d065a-5535-41fb-bbc6-ba2fc54cd0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 29 22:51:31.797140 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:31.797107    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"
Oct 29 22:51:32.114943 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:32.113774    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq" event={"ID":"d190385b-94ea-4b39-9318-3a33e14e8286","Type":"ContainerStarted","Data":"6f11f03f3179b369d195ccfa03394f3f21a5e21315a56272e5d455f1abf2a78d"}
Oct 29 22:51:32.114943 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:32.113864    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq" event={"ID":"d190385b-94ea-4b39-9318-3a33e14e8286","Type":"ContainerStarted","Data":"e02af42845c09ad219966e2200433582ddc9c185d78e9a288ae64e9cc8560ff0"}
Oct 29 22:51:32.114943 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:32.113913    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"
Oct 29 22:51:32.157282 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:32.157204    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq" podStartSLOduration=2.157188207 podStartE2EDuration="2.157188207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:32.153439975 +0000 UTC m=+2231.064883529" watchObservedRunningTime="2026-10-29 22:51:32.157188207 +0000 UTC m=+2231.068631761"
Oct 29 22:51:32.518202 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:32.518158    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:51:32.520932 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:32.520863    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:51:32.527010 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:32.526979    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 22:51:32.647269 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:32.647211    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:51:32.648436 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:32.648411    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 22:51:43.126087 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:43.126057    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"
Oct 29 22:51:43.696287 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:51:43.696247    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:51:47.513077 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.513041    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"]
Oct 29 22:51:47.519143 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.519111    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.538102 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.538063    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c87bf1-openshift-default-dockercfg-njcgb\""
Oct 29 22:51:47.558085 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.558046    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"]
Oct 29 22:51:47.574115 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.574073    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-podinfo\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.574335 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.574223    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-workload-certs\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.574335 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.574262    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-workload-socket\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.574475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.574357    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-data\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.574475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.574389    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-token\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.574475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.574444    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istiod-ca-cert\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.574669 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.574485    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhgnc\" (UniqueName: \"kubernetes.io/projected/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-kube-api-access-vhgnc\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.574669 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.574521    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-credential-socket\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.574669 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.574604    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-envoy\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.694837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.681948    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-envoy\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.694837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.682078    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-podinfo\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.694837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.682199    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-workload-certs\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.694837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.682233    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-workload-socket\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.694837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.682326    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-data\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.694837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.682354    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-token\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.694837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.682413    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istiod-ca-cert\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.694837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.682452    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhgnc\" (UniqueName: \"kubernetes.io/projected/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-kube-api-access-vhgnc\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.694837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.682489    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-credential-socket\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.694837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.684057    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-credential-socket\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.694837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.687414    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-workload-certs\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.694837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.687753    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-workload-socket\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.694837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.688044    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-data\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.694837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.688986    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istiod-ca-cert\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.726839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.707305    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-envoy\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.728603 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.728568    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-podinfo\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.759843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.753463    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-token\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.787891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.777828    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhgnc\" (UniqueName: \"kubernetes.io/projected/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-kube-api-access-vhgnc\") pod \"e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") " pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:47.853835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:47.853372    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:48.255270 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:48.255240    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"]
Oct 29 22:51:48.258187 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:48.258147    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:51:48.258313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:48.258226    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:51:48.258313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:48.258257    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:51:48.269523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:48.269494    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t" event={"ID":"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2","Type":"ContainerStarted","Data":"a016d05dfc43702c7e58b479da42cbf635abdb2dfb0d78140b780979505820c1"}
Oct 29 22:51:49.280326 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:49.280287    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t" event={"ID":"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2","Type":"ContainerStarted","Data":"c6dcb7da1747cd025dc7d1a33d3277807b905d2546f0193e501f937c13241070"}
Oct 29 22:51:49.334794 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:49.334738    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t" podStartSLOduration=2.3347203739999998 podStartE2EDuration="2.334720374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:49.334278938 +0000 UTC m=+2248.245722504" watchObservedRunningTime="2026-10-29 22:51:49.334720374 +0000 UTC m=+2248.246163930"
Oct 29 22:51:49.853668 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:49.853619    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:49.860780 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:49.860745    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:50.287870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:50.287762    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:50.288977 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:50.288948    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:51:53.611496 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:53.611459    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"]
Oct 29 22:51:53.618239 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:53.618211    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"
Oct 29 22:51:53.643422 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:53.643386    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"]
Oct 29 22:51:53.803855 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:53.803783    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c87bf1-dockercfg-jvlkf\""
Oct 29 22:51:53.807261 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:53.807231    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c87bf1\""
Oct 29 22:51:53.820839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:53.808757    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvcdg\" (UniqueName: \"kubernetes.io/projected/8c02048e-7f83-460e-8f29-f84a6717b37f-kube-api-access-gvcdg\") pod \"payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm\" (UID: \"8c02048e-7f83-460e-8f29-f84a6717b37f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"
Oct 29 22:51:53.821205 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:53.821173    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c02048e-7f83-460e-8f29-f84a6717b37f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm\" (UID: \"8c02048e-7f83-460e-8f29-f84a6717b37f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"
Oct 29 22:51:53.972845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:53.972089    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcdg\" (UniqueName: \"kubernetes.io/projected/8c02048e-7f83-460e-8f29-f84a6717b37f-kube-api-access-gvcdg\") pod \"payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm\" (UID: \"8c02048e-7f83-460e-8f29-f84a6717b37f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"
Oct 29 22:51:53.972845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:53.972182    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c02048e-7f83-460e-8f29-f84a6717b37f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm\" (UID: \"8c02048e-7f83-460e-8f29-f84a6717b37f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"
Oct 29 22:51:53.978383 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:53.973269    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c02048e-7f83-460e-8f29-f84a6717b37f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm\" (UID: \"8c02048e-7f83-460e-8f29-f84a6717b37f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"
Oct 29 22:51:53.992760 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:53.992713    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"]
Oct 29 22:51:53.999203 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:53.999171    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"
Oct 29 22:51:54.005285 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:54.004976    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvcdg\" (UniqueName: \"kubernetes.io/projected/8c02048e-7f83-460e-8f29-f84a6717b37f-kube-api-access-gvcdg\") pod \"payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm\" (UID: \"8c02048e-7f83-460e-8f29-f84a6717b37f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"
Oct 29 22:51:54.006773 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:54.006734    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"]
Oct 29 22:51:54.199353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:54.176110    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqld5\" (UniqueName: \"kubernetes.io/projected/ab9dbe77-9888-4a77-a333-68c2899fdb65-kube-api-access-bqld5\") pod \"payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk\" (UID: \"ab9dbe77-9888-4a77-a333-68c2899fdb65\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"
Oct 29 22:51:54.199353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:54.176333    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab9dbe77-9888-4a77-a333-68c2899fdb65-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk\" (UID: \"ab9dbe77-9888-4a77-a333-68c2899fdb65\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"
Oct 29 22:51:54.248195 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:54.248112    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"
Oct 29 22:51:54.277631 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:54.277577    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqld5\" (UniqueName: \"kubernetes.io/projected/ab9dbe77-9888-4a77-a333-68c2899fdb65-kube-api-access-bqld5\") pod \"payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk\" (UID: \"ab9dbe77-9888-4a77-a333-68c2899fdb65\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"
Oct 29 22:51:54.278011 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:54.277972    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab9dbe77-9888-4a77-a333-68c2899fdb65-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk\" (UID: \"ab9dbe77-9888-4a77-a333-68c2899fdb65\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"
Oct 29 22:51:54.278935 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:54.278908    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab9dbe77-9888-4a77-a333-68c2899fdb65-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk\" (UID: \"ab9dbe77-9888-4a77-a333-68c2899fdb65\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"
Oct 29 22:51:54.297880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:54.297551    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqld5\" (UniqueName: \"kubernetes.io/projected/ab9dbe77-9888-4a77-a333-68c2899fdb65-kube-api-access-bqld5\") pod \"payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk\" (UID: \"ab9dbe77-9888-4a77-a333-68c2899fdb65\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"
Oct 29 22:51:54.351251 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:54.344105    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"
Oct 29 22:51:54.633798 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:54.633762    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"]
Oct 29 22:51:54.634486 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:51:54.634457    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c02048e_7f83_460e_8f29_f84a6717b37f.slice/crio-92447feb4ce41850b7c85d4ad38fdcab372060de61fecc01ac8c0afc44d0ea3b WatchSource:0}: Error finding container 92447feb4ce41850b7c85d4ad38fdcab372060de61fecc01ac8c0afc44d0ea3b: Status 404 returned error can't find the container with id 92447feb4ce41850b7c85d4ad38fdcab372060de61fecc01ac8c0afc44d0ea3b
Oct 29 22:51:54.749251 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:51:54.749216    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9dbe77_9888_4a77_a333_68c2899fdb65.slice/crio-b9dbd5832a6a4b77492909980e87bdf3ae36774703428f460d0b47548cf5d06c WatchSource:0}: Error finding container b9dbd5832a6a4b77492909980e87bdf3ae36774703428f460d0b47548cf5d06c: Status 404 returned error can't find the container with id b9dbd5832a6a4b77492909980e87bdf3ae36774703428f460d0b47548cf5d06c
Oct 29 22:51:54.750379 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:54.750317    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"]
Oct 29 22:51:55.345489 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:55.345449    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm" event={"ID":"8c02048e-7f83-460e-8f29-f84a6717b37f","Type":"ContainerStarted","Data":"41231dc40c6af79efdccee9504df9756e426fee4415faa2b0482ba319dc57950"}
Oct 29 22:51:55.345679 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:55.345510    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm" event={"ID":"8c02048e-7f83-460e-8f29-f84a6717b37f","Type":"ContainerStarted","Data":"92447feb4ce41850b7c85d4ad38fdcab372060de61fecc01ac8c0afc44d0ea3b"}
Oct 29 22:51:55.345679 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:55.345559    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"
Oct 29 22:51:55.347471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:55.347443    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk" event={"ID":"ab9dbe77-9888-4a77-a333-68c2899fdb65","Type":"ContainerStarted","Data":"8127b5192efe570debcf87800b7f077ba208dd8b89da6779e382db4aa48d6258"}
Oct 29 22:51:55.347651 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:55.347499    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk" event={"ID":"ab9dbe77-9888-4a77-a333-68c2899fdb65","Type":"ContainerStarted","Data":"b9dbd5832a6a4b77492909980e87bdf3ae36774703428f460d0b47548cf5d06c"}
Oct 29 22:51:55.347651 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:55.347538    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"
Oct 29 22:51:55.386055 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:55.385998    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm" podStartSLOduration=2.385982974 podStartE2EDuration="2.385982974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:55.385063788 +0000 UTC m=+2254.296507341" watchObservedRunningTime="2026-10-29 22:51:55.385982974 +0000 UTC m=+2254.297426528"
Oct 29 22:51:55.428209 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:51:55.428154    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk" podStartSLOduration=2.428139159 podStartE2EDuration="2.428139159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:55.423889534 +0000 UTC m=+2254.335333089" watchObservedRunningTime="2026-10-29 22:51:55.428139159 +0000 UTC m=+2254.339582713"
Oct 29 22:52:06.361659 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:06.361621    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"
Oct 29 22:52:06.362261 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:06.361719    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"
Oct 29 22:52:10.535159 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:10.535124    4559 generic.go:356] "Generic (PLEG): container finished" podID="bde5a64f-e410-46b8-8e6f-c2933360770a" containerID="4a2e8cf6f91c20c461350b7aad28501e7d95c5c1389a463b17cbfb527e920529" exitCode=0
Oct 29 22:52:10.535913 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:10.535790    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl" event={"ID":"bde5a64f-e410-46b8-8e6f-c2933360770a","Type":"ContainerDied","Data":"4a2e8cf6f91c20c461350b7aad28501e7d95c5c1389a463b17cbfb527e920529"}
Oct 29 22:52:10.536607 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:10.536586    4559 scope.go:122] "RemoveContainer" containerID="4a2e8cf6f91c20c461350b7aad28501e7d95c5c1389a463b17cbfb527e920529"
Oct 29 22:52:10.547207 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:10.547167    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:52:10.547331 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:10.547287    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:52:10.547376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:10.547336    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:52:11.141318 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:11.141276    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:52:11.569385 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:11.569285    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl" event={"ID":"bde5a64f-e410-46b8-8e6f-c2933360770a","Type":"ContainerStarted","Data":"9ea104efd748881e5a4ff0e329b2447f90218ad74d4f0fc5db7e2cda9a6813e3"}
Oct 29 22:52:12.309916 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:12.309878    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:52:12.311605 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:12.311578    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:52:12.311731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:12.311626    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl" podUID="bde5a64f-e410-46b8-8e6f-c2933360770a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:52:13.309673 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:13.309637    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:52:13.310169 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:13.309706    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl" podUID="bde5a64f-e410-46b8-8e6f-c2933360770a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:52:13.698728 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:52:13.698691    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:52:14.309170 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:14.309140    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:52:14.309369 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:14.309198    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl" podUID="bde5a64f-e410-46b8-8e6f-c2933360770a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:52:15.310111 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:15.310076    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:52:15.310616 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:15.310143    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl" podUID="bde5a64f-e410-46b8-8e6f-c2933360770a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:52:16.310076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:16.310040    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:52:16.310275 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:16.310107    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl" podUID="bde5a64f-e410-46b8-8e6f-c2933360770a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:52:17.314686 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:17.314655    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:52:17.315161 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:17.315147    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:52:17.315991 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:17.315973    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 22:52:38.465944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.465857    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"]
Oct 29 22:52:38.473769 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.473739    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.489523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.489495    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-920032\"/\"openshift-service-ca.crt\""
Oct 29 22:52:38.489675 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.489520    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-920032\"/\"kube-root-ca.crt\""
Oct 29 22:52:38.492171 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.492142    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"]
Oct 29 22:52:38.498313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.498247    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-920032\"/\"default-dockercfg-6br9l\""
Oct 29 22:52:38.498313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.498277    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-920032\"/\"auth-test-model-920032-kserve-self-signed-certs\""
Oct 29 22:52:38.623092 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.623051    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/882dc67e-fe87-4033-8be0-bfdc48845531-tls-certs\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.623302 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.623108    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcxnk\" (UniqueName: \"kubernetes.io/projected/882dc67e-fe87-4033-8be0-bfdc48845531-kube-api-access-jcxnk\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.623302 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.623187    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-home\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.623302 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.623218    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-dshm\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.623302 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.623237    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-tmp-dir\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.623522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.623377    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-model-cache\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.725890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.724801    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-model-cache\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.725890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.725136    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/882dc67e-fe87-4033-8be0-bfdc48845531-tls-certs\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.725890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.725209    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcxnk\" (UniqueName: \"kubernetes.io/projected/882dc67e-fe87-4033-8be0-bfdc48845531-kube-api-access-jcxnk\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.725890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.725272    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-home\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.725890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.725317    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-dshm\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.725890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.725348    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-tmp-dir\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.726353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.726095    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-tmp-dir\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.726421 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.726351    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-home\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.734846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.726599    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-model-cache\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.765832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.761409    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/882dc67e-fe87-4033-8be0-bfdc48845531-tls-certs\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.765832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.764589    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcxnk\" (UniqueName: \"kubernetes.io/projected/882dc67e-fe87-4033-8be0-bfdc48845531-kube-api-access-jcxnk\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.781835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.775277    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-dshm\") pod \"auth-test-model-920032-kserve-6d7cd9d6df-zqkc8\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") " pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:38.794439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:38.793127    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:39.181798 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:52:39.181763    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod882dc67e_fe87_4033_8be0_bfdc48845531.slice/crio-cf220609dfd6fadfd6172d2622fd373f850c3e16be13821a47aeabddbf344e1a WatchSource:0}: Error finding container cf220609dfd6fadfd6172d2622fd373f850c3e16be13821a47aeabddbf344e1a: Status 404 returned error can't find the container with id cf220609dfd6fadfd6172d2622fd373f850c3e16be13821a47aeabddbf344e1a
Oct 29 22:52:39.182495 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:39.182468    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"]
Oct 29 22:52:39.866608 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:39.866570    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8" event={"ID":"882dc67e-fe87-4033-8be0-bfdc48845531","Type":"ContainerStarted","Data":"201536958e6ab2e917cc2390560277bee542945baa4d703a34fc3f7e9d3b9a79"}
Oct 29 22:52:39.867160 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:39.866618    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8" event={"ID":"882dc67e-fe87-4033-8be0-bfdc48845531","Type":"ContainerStarted","Data":"cf220609dfd6fadfd6172d2622fd373f850c3e16be13821a47aeabddbf344e1a"}
Oct 29 22:52:39.908517 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:39.908461    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8" podStartSLOduration=1.908445034 podStartE2EDuration="1.908445034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:39.907231268 +0000 UTC m=+2298.818674846" watchObservedRunningTime="2026-10-29 22:52:39.908445034 +0000 UTC m=+2298.819888587"
Oct 29 22:52:40.794524 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:40.794484    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:41.755166 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:41.755130    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:52:43.918150 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:52:43.918109    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:52:48.800831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:48.799467    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:48.935783 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:48.935749    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:49.083560 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:49.083470    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:52:54.527509 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.527468    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"]
Oct 29 22:52:54.532634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.532605    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.569206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.569122    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"]
Oct 29 22:52:54.636968 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.636909    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.637163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.636977    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.637163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.637038    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.637292 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.637161    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc64k\" (UniqueName: \"kubernetes.io/projected/84e86523-8b8b-446d-a5de-25172d07da2d-kube-api-access-xc64k\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.637292 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.637245    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.637392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.637289    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e86523-8b8b-446d-a5de-25172d07da2d-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.637433 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.637400    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e86523-8b8b-446d-a5de-25172d07da2d-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.637586 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.637568    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.637627 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.637605    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e86523-8b8b-446d-a5de-25172d07da2d-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.739343 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.739294    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.739567 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.739369    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e86523-8b8b-446d-a5de-25172d07da2d-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.739567 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.739419    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e86523-8b8b-446d-a5de-25172d07da2d-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.739567 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.739560    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.739760 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.739591    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e86523-8b8b-446d-a5de-25172d07da2d-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.739849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.739760    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.739849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.739800    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.739964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.739877    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.739964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.739901    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.739964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.739951    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc64k\" (UniqueName: \"kubernetes.io/projected/84e86523-8b8b-446d-a5de-25172d07da2d-kube-api-access-xc64k\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.740111 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.740090    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.740487 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.740461    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e86523-8b8b-446d-a5de-25172d07da2d-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.740591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.740521    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.740902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.740619    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.742470 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.742441    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.742676 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.742659    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e86523-8b8b-446d-a5de-25172d07da2d-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.759206 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.759179    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e86523-8b8b-446d-a5de-25172d07da2d-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.759458 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.759433    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc64k\" (UniqueName: \"kubernetes.io/projected/84e86523-8b8b-446d-a5de-25172d07da2d-kube-api-access-xc64k\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:54.851853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:54.851738    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:55.199429 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:55.199393    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"]
Oct 29 22:52:55.203123 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:52:55.203086    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e86523_8b8b_446d_a5de_25172d07da2d.slice/crio-0a497f066bf2396e229772f63d58eb4510cb649bdf919668af120f4b4be0b682 WatchSource:0}: Error finding container 0a497f066bf2396e229772f63d58eb4510cb649bdf919668af120f4b4be0b682: Status 404 returned error can't find the container with id 0a497f066bf2396e229772f63d58eb4510cb649bdf919668af120f4b4be0b682
Oct 29 22:52:55.206610 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:55.206570    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:52:55.206740 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:55.206648    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:52:55.206740 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:55.206680    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:52:55.233613 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:55.233559    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd" event={"ID":"84e86523-8b8b-446d-a5de-25172d07da2d","Type":"ContainerStarted","Data":"0a497f066bf2396e229772f63d58eb4510cb649bdf919668af120f4b4be0b682"}
Oct 29 22:52:56.249049 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:56.249008    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd" event={"ID":"84e86523-8b8b-446d-a5de-25172d07da2d","Type":"ContainerStarted","Data":"8f51657188e0a12a14e3e6b10ebcb1b1639e4671e03e599b7860e09ee82a10d8"}
Oct 29 22:52:56.367842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:56.332686    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd" podStartSLOduration=2.332666408 podStartE2EDuration="2.332666408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:56.315339848 +0000 UTC m=+2315.226783403" watchObservedRunningTime="2026-10-29 22:52:56.332666408 +0000 UTC m=+2315.244109963"
Oct 29 22:52:56.856835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:56.855875    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:52:56.863847 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:56.862954    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 29 22:52:56.863847 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:56.863035    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd" podUID="84e86523-8b8b-446d-a5de-25172d07da2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 29 22:52:57.857840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:57.855287    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 29 22:52:57.857840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:57.855355    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd" podUID="84e86523-8b8b-446d-a5de-25172d07da2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 29 22:52:58.861836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:58.861124    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 29 22:52:58.861836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:58.861186    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd" podUID="84e86523-8b8b-446d-a5de-25172d07da2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 29 22:52:59.860837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:59.852253    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 29 22:52:59.860837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:52:59.852360    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd" podUID="84e86523-8b8b-446d-a5de-25172d07da2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 29 22:53:00.859836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:00.852217    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 29 22:53:00.859836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:00.852295    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd" podUID="84e86523-8b8b-446d-a5de-25172d07da2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 29 22:53:01.864834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:01.861306    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 29 22:53:01.864834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:01.861390    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd" podUID="84e86523-8b8b-446d-a5de-25172d07da2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 29 22:53:02.857658 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:02.857620    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:53:02.858107 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:02.858085    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:53:02.858994 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:02.858974    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 22:53:13.705010 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:53:13.704961    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:53:18.510370 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:18.510332    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"]
Oct 29 22:53:18.560907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:18.560868    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"]
Oct 29 22:53:21.329976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:21.329934    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:53:21.330569 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:21.330003    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t" podUID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:23.514951 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:23.512571    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk" podUID="ab9dbe77-9888-4a77-a333-68c2899fdb65" containerName="payload-processing" containerID="cri-o://8127b5192efe570debcf87800b7f077ba208dd8b89da6779e382db4aa48d6258" gracePeriod=25
Oct 29 22:53:23.561829 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:23.561758    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm" podUID="8c02048e-7f83-460e-8f29-f84a6717b37f" containerName="payload-pre-processing" containerID="cri-o://41231dc40c6af79efdccee9504df9756e426fee4415faa2b0482ba319dc57950" gracePeriod=25
Oct 29 22:53:23.648449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:23.648410    4559 generic.go:356] "Generic (PLEG): container finished" podID="ab9dbe77-9888-4a77-a333-68c2899fdb65" containerID="8127b5192efe570debcf87800b7f077ba208dd8b89da6779e382db4aa48d6258" exitCode=0
Oct 29 22:53:23.648647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:23.648486    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk" event={"ID":"ab9dbe77-9888-4a77-a333-68c2899fdb65","Type":"ContainerDied","Data":"8127b5192efe570debcf87800b7f077ba208dd8b89da6779e382db4aa48d6258"}
Oct 29 22:53:23.909761 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:23.909734    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"
Oct 29 22:53:23.938225 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:23.938200    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"
Oct 29 22:53:24.001426 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.001395    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab9dbe77-9888-4a77-a333-68c2899fdb65-kube-api-access-bqld5\" (UniqueName: \"kubernetes.io/projected/ab9dbe77-9888-4a77-a333-68c2899fdb65-kube-api-access-bqld5\") pod \"ab9dbe77-9888-4a77-a333-68c2899fdb65\" (UID: \"ab9dbe77-9888-4a77-a333-68c2899fdb65\") "
Oct 29 22:53:24.001699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.001676    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab9dbe77-9888-4a77-a333-68c2899fdb65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab9dbe77-9888-4a77-a333-68c2899fdb65-plugins-config-volume\") pod \"ab9dbe77-9888-4a77-a333-68c2899fdb65\" (UID: \"ab9dbe77-9888-4a77-a333-68c2899fdb65\") "
Oct 29 22:53:24.002550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.002521    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab9dbe77-9888-4a77-a333-68c2899fdb65-plugins-config-volume" pod "ab9dbe77-9888-4a77-a333-68c2899fdb65" (UID: "ab9dbe77-9888-4a77-a333-68c2899fdb65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:53:24.004606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.004581    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9dbe77-9888-4a77-a333-68c2899fdb65-kube-api-access-bqld5" pod "ab9dbe77-9888-4a77-a333-68c2899fdb65" (UID: "ab9dbe77-9888-4a77-a333-68c2899fdb65"). InnerVolumeSpecName "kube-api-access-bqld5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:53:24.103887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.103778    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c02048e-7f83-460e-8f29-f84a6717b37f-kube-api-access-gvcdg\" (UniqueName: \"kubernetes.io/projected/8c02048e-7f83-460e-8f29-f84a6717b37f-kube-api-access-gvcdg\") pod \"8c02048e-7f83-460e-8f29-f84a6717b37f\" (UID: \"8c02048e-7f83-460e-8f29-f84a6717b37f\") "
Oct 29 22:53:24.104155 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.104135    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c02048e-7f83-460e-8f29-f84a6717b37f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c02048e-7f83-460e-8f29-f84a6717b37f-plugins-config-volume\") pod \"8c02048e-7f83-460e-8f29-f84a6717b37f\" (UID: \"8c02048e-7f83-460e-8f29-f84a6717b37f\") "
Oct 29 22:53:24.104488 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.104463    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c02048e-7f83-460e-8f29-f84a6717b37f-plugins-config-volume" pod "8c02048e-7f83-460e-8f29-f84a6717b37f" (UID: "8c02048e-7f83-460e-8f29-f84a6717b37f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:53:24.104654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.104637    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqld5\" (UniqueName: \"kubernetes.io/projected/ab9dbe77-9888-4a77-a333-68c2899fdb65-kube-api-access-bqld5\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:53:24.104727 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.104659    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab9dbe77-9888-4a77-a333-68c2899fdb65-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:53:24.104727 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.104673    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c02048e-7f83-460e-8f29-f84a6717b37f-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:53:24.107036 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.107015    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c02048e-7f83-460e-8f29-f84a6717b37f-kube-api-access-gvcdg" pod "8c02048e-7f83-460e-8f29-f84a6717b37f" (UID: "8c02048e-7f83-460e-8f29-f84a6717b37f"). InnerVolumeSpecName "kube-api-access-gvcdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:53:24.206089 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.206041    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvcdg\" (UniqueName: \"kubernetes.io/projected/8c02048e-7f83-460e-8f29-f84a6717b37f-kube-api-access-gvcdg\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:53:24.657613 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.657579    4559 generic.go:356] "Generic (PLEG): container finished" podID="8c02048e-7f83-460e-8f29-f84a6717b37f" containerID="41231dc40c6af79efdccee9504df9756e426fee4415faa2b0482ba319dc57950" exitCode=0
Oct 29 22:53:24.658185 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.657667    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm" event={"ID":"8c02048e-7f83-460e-8f29-f84a6717b37f","Type":"ContainerDied","Data":"41231dc40c6af79efdccee9504df9756e426fee4415faa2b0482ba319dc57950"}
Oct 29 22:53:24.658185 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.657719    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"
Oct 29 22:53:24.658185 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.657741    4559 scope.go:122] "RemoveContainer" containerID="41231dc40c6af79efdccee9504df9756e426fee4415faa2b0482ba319dc57950"
Oct 29 22:53:24.658185 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.657727    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm" event={"ID":"8c02048e-7f83-460e-8f29-f84a6717b37f","Type":"ContainerDied","Data":"92447feb4ce41850b7c85d4ad38fdcab372060de61fecc01ac8c0afc44d0ea3b"}
Oct 29 22:53:24.659842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.659820    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"
Oct 29 22:53:24.660100 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.659929    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk" event={"ID":"ab9dbe77-9888-4a77-a333-68c2899fdb65","Type":"ContainerDied","Data":"b9dbd5832a6a4b77492909980e87bdf3ae36774703428f460d0b47548cf5d06c"}
Oct 29 22:53:24.663601 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.663568    4559 generic.go:356] "Generic (PLEG): container finished" podID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" containerID="c6dcb7da1747cd025dc7d1a33d3277807b905d2546f0193e501f937c13241070" exitCode=0
Oct 29 22:53:24.663728 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.663696    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t" event={"ID":"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2","Type":"ContainerDied","Data":"c6dcb7da1747cd025dc7d1a33d3277807b905d2546f0193e501f937c13241070"}
Oct 29 22:53:24.664273 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.664251    4559 scope.go:122] "RemoveContainer" containerID="c6dcb7da1747cd025dc7d1a33d3277807b905d2546f0193e501f937c13241070"
Oct 29 22:53:24.666584 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.666548    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:53:24.666691 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.666642    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:53:24.666752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.666688    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:53:24.675422 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.675404    4559 scope.go:122] "RemoveContainer" containerID="41231dc40c6af79efdccee9504df9756e426fee4415faa2b0482ba319dc57950"
Oct 29 22:53:24.677855 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:53:24.676020    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41231dc40c6af79efdccee9504df9756e426fee4415faa2b0482ba319dc57950\": container with ID starting with 41231dc40c6af79efdccee9504df9756e426fee4415faa2b0482ba319dc57950 not found: ID does not exist" containerID="41231dc40c6af79efdccee9504df9756e426fee4415faa2b0482ba319dc57950"
Oct 29 22:53:24.677855 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.676053    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41231dc40c6af79efdccee9504df9756e426fee4415faa2b0482ba319dc57950"} err="failed to get container status \"41231dc40c6af79efdccee9504df9756e426fee4415faa2b0482ba319dc57950\": rpc error: code = NotFound desc = could not find container \"41231dc40c6af79efdccee9504df9756e426fee4415faa2b0482ba319dc57950\": container with ID starting with 41231dc40c6af79efdccee9504df9756e426fee4415faa2b0482ba319dc57950 not found: ID does not exist"
Oct 29 22:53:24.677855 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.676075    4559 scope.go:122] "RemoveContainer" containerID="8127b5192efe570debcf87800b7f077ba208dd8b89da6779e382db4aa48d6258"
Oct 29 22:53:24.712831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.712779    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"]
Oct 29 22:53:24.719136 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.719104    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c87bf1-5cf8c8df7d-xwbvk"]
Oct 29 22:53:24.961905 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.960473    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"]
Oct 29 22:53:24.962654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:24.962622    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c87bf1-84c7b6485f-cxhjm"]
Oct 29 22:53:25.285837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:25.274667    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c02048e-7f83-460e-8f29-f84a6717b37f" path="/var/lib/kubelet/pods/8c02048e-7f83-460e-8f29-f84a6717b37f/volumes"
Oct 29 22:53:25.287575 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:25.287532    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab9dbe77-9888-4a77-a333-68c2899fdb65" path="/var/lib/kubelet/pods/ab9dbe77-9888-4a77-a333-68c2899fdb65/volumes"
Oct 29 22:53:25.685921 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:25.685887    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t" event={"ID":"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2","Type":"ContainerStarted","Data":"47d0a75cc39ffcafd7635f4c3e1e340478424350790f9a3d14c1d615c83c3ef1"}
Oct 29 22:53:25.854392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:25.854351    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:53:25.854599 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:25.854529    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 29 22:53:25.854599 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:25.854571    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t" podUID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 29 22:53:26.856395 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:26.856349    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 29 22:53:26.857164 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:26.857129    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t" podUID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 29 22:53:27.856836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:27.856289    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 29 22:53:27.856836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:27.856372    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t" podUID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 29 22:53:28.863835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:28.863256    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 29 22:53:28.863835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:28.863348    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t" podUID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 29 22:53:29.613945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.613905    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"]
Oct 29 22:53:29.615124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.615100    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ab9dbe77-9888-4a77-a333-68c2899fdb65" containerName="payload-processing"
Oct 29 22:53:29.615124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.615123    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab9dbe77-9888-4a77-a333-68c2899fdb65" containerName="payload-processing"
Oct 29 22:53:29.615320 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.615173    4559 cpu_manager.go:414] "containerMap: removing container" podUID="8c02048e-7f83-460e-8f29-f84a6717b37f" containerName="payload-pre-processing"
Oct 29 22:53:29.615320 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.615181    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c02048e-7f83-460e-8f29-f84a6717b37f" containerName="payload-pre-processing"
Oct 29 22:53:29.615444 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.615377    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab9dbe77-9888-4a77-a333-68c2899fdb65" containerName="payload-processing"
Oct 29 22:53:29.615444 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.615401    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c02048e-7f83-460e-8f29-f84a6717b37f" containerName="payload-pre-processing"
Oct 29 22:53:29.620665 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.620639    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.677907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.660312    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-02720a\"/\"default-dockercfg-6srqf\""
Oct 29 22:53:29.677907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.660407    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-02720a\"/\"openshift-service-ca.crt\""
Oct 29 22:53:29.677907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.660970    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-02720a\"/\"kube-root-ca.crt\""
Oct 29 22:53:29.677907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.673019    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-02720a\"/\"auth-test-model-02720a-kserve-self-signed-certs\""
Oct 29 22:53:29.677907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.676928    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"]
Oct 29 22:53:29.769329 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.769294    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-tls-certs\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.769744 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.769719    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jphv7\" (UniqueName: \"kubernetes.io/projected/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-kube-api-access-jphv7\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.770006 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.769973    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-home\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.770151 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.770098    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-tmp-dir\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.770498 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.770474    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-dshm\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.770628 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.770612    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-model-cache\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.864332 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.864242    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 29 22:53:29.864925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.864892    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t" podUID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 29 22:53:29.884484 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.884436    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-tls-certs\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.884853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.884833    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jphv7\" (UniqueName: \"kubernetes.io/projected/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-kube-api-access-jphv7\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.885101 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.885082    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-home\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.911720 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.911688    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-tmp-dir\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.912118 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.912094    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-dshm\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.912347 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.912330    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-model-cache\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.920858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.912951    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-model-cache\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.921118 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.914707    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-tmp-dir\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.921230 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.894284    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-home\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.921320 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.904115    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-tls-certs\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.926873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.926841    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-dshm\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.933405 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.933368    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jphv7\" (UniqueName: \"kubernetes.io/projected/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-kube-api-access-jphv7\") pod \"auth-test-model-02720a-kserve-54bc7f774f-nz6fz\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") " pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:29.979672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:29.979629    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:30.470837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:30.465023    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"]
Oct 29 22:53:30.470837 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:53:30.466926    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ee1f4f6_4924_4dfd_8bab_af2c55dd4c93.slice/crio-5da072a4b5ab2d883e3b9f2b4a235cb2074fef5c20d3476988db36463bd414c7 WatchSource:0}: Error finding container 5da072a4b5ab2d883e3b9f2b4a235cb2074fef5c20d3476988db36463bd414c7: Status 404 returned error can't find the container with id 5da072a4b5ab2d883e3b9f2b4a235cb2074fef5c20d3476988db36463bd414c7
Oct 29 22:53:30.790656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:30.790558    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz" event={"ID":"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93","Type":"ContainerStarted","Data":"b83d4441342378ff1d249b25e60fb230cd218efd9d57e6ac859c12abef535803"}
Oct 29 22:53:30.790656 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:30.790610    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz" event={"ID":"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93","Type":"ContainerStarted","Data":"5da072a4b5ab2d883e3b9f2b4a235cb2074fef5c20d3476988db36463bd414c7"}
Oct 29 22:53:30.855836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:30.838792    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz" podStartSLOduration=1.838770941 podStartE2EDuration="1.838770941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:30.829614483 +0000 UTC m=+2349.741058037" watchObservedRunningTime="2026-10-29 22:53:30.838770941 +0000 UTC m=+2349.750214496"
Oct 29 22:53:30.859833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:30.859250    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 29 22:53:30.859833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:30.859314    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t" podUID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 29 22:53:30.981849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:30.980959    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:31.867571 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:31.867538    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:53:31.869134 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:31.869106    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:53:31.870457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:31.870422    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:53:38.776123 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:38.776071    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"]
Oct 29 22:53:40.120873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:40.097785    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:40.294988 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:40.294954    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:40.506948 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:40.506908    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:53:40.992728 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:40.992694    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:53:43.338703 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:43.337931    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:43.671851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:43.669933    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:43.760194 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:53:43.760125    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:53:46.370072 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.370030    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"]
Oct 29 22:53:46.375410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.375383    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"
Oct 29 22:53:46.396574 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.396534    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"]
Oct 29 22:53:46.415377 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.415333    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"]
Oct 29 22:53:46.515846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.506740    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7be71d-dockercfg-zp9qt\""
Oct 29 22:53:46.515846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.506887    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7be71d\""
Oct 29 22:53:46.581840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.581234    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"
Oct 29 22:53:46.675912 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.657095    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"]
Oct 29 22:53:46.787032 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.786988    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fce30c-5c90-464b-9881-fa411cb2c320-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5\" (UID: \"89fce30c-5c90-464b-9881-fa411cb2c320\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"
Oct 29 22:53:46.787241 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.787223    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dnvg\" (UniqueName: \"kubernetes.io/projected/dacadeaa-93d8-43fe-a074-5d3b4ef8eae6-kube-api-access-6dnvg\") pod \"payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm\" (UID: \"dacadeaa-93d8-43fe-a074-5d3b4ef8eae6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"
Oct 29 22:53:46.787291 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.787273    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv5q4\" (UniqueName: \"kubernetes.io/projected/89fce30c-5c90-464b-9881-fa411cb2c320-kube-api-access-qv5q4\") pod \"payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5\" (UID: \"89fce30c-5c90-464b-9881-fa411cb2c320\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"
Oct 29 22:53:46.787388 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.787368    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacadeaa-93d8-43fe-a074-5d3b4ef8eae6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm\" (UID: \"dacadeaa-93d8-43fe-a074-5d3b4ef8eae6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"
Oct 29 22:53:46.889050 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.889002    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fce30c-5c90-464b-9881-fa411cb2c320-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5\" (UID: \"89fce30c-5c90-464b-9881-fa411cb2c320\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"
Oct 29 22:53:46.889280 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.889148    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnvg\" (UniqueName: \"kubernetes.io/projected/dacadeaa-93d8-43fe-a074-5d3b4ef8eae6-kube-api-access-6dnvg\") pod \"payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm\" (UID: \"dacadeaa-93d8-43fe-a074-5d3b4ef8eae6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"
Oct 29 22:53:46.889280 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.889187    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv5q4\" (UniqueName: \"kubernetes.io/projected/89fce30c-5c90-464b-9881-fa411cb2c320-kube-api-access-qv5q4\") pod \"payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5\" (UID: \"89fce30c-5c90-464b-9881-fa411cb2c320\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"
Oct 29 22:53:46.889280 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.889218    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacadeaa-93d8-43fe-a074-5d3b4ef8eae6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm\" (UID: \"dacadeaa-93d8-43fe-a074-5d3b4ef8eae6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"
Oct 29 22:53:46.889995 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.889966    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fce30c-5c90-464b-9881-fa411cb2c320-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5\" (UID: \"89fce30c-5c90-464b-9881-fa411cb2c320\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"
Oct 29 22:53:46.890073 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.890057    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacadeaa-93d8-43fe-a074-5d3b4ef8eae6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm\" (UID: \"dacadeaa-93d8-43fe-a074-5d3b4ef8eae6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"
Oct 29 22:53:46.909835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.909779    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv5q4\" (UniqueName: \"kubernetes.io/projected/89fce30c-5c90-464b-9881-fa411cb2c320-kube-api-access-qv5q4\") pod \"payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5\" (UID: \"89fce30c-5c90-464b-9881-fa411cb2c320\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"
Oct 29 22:53:46.909990 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.909788    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dnvg\" (UniqueName: \"kubernetes.io/projected/dacadeaa-93d8-43fe-a074-5d3b4ef8eae6-kube-api-access-6dnvg\") pod \"payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm\" (UID: \"dacadeaa-93d8-43fe-a074-5d3b4ef8eae6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"
Oct 29 22:53:46.994790 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:46.994697    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"
Oct 29 22:53:47.059296 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:47.059256    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"
Oct 29 22:53:47.330220 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:47.330017    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"]
Oct 29 22:53:47.332609 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:53:47.332574    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacadeaa_93d8_43fe_a074_5d3b4ef8eae6.slice/crio-14910bb220a27678b1148f65b03e429ba9b48f55d0bc8f9507bb430997f23266 WatchSource:0}: Error finding container 14910bb220a27678b1148f65b03e429ba9b48f55d0bc8f9507bb430997f23266: Status 404 returned error can't find the container with id 14910bb220a27678b1148f65b03e429ba9b48f55d0bc8f9507bb430997f23266
Oct 29 22:53:47.414621 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:47.414588    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"]
Oct 29 22:53:47.415079 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:53:47.415050    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fce30c_5c90_464b_9881_fa411cb2c320.slice/crio-16ad3f4a4679f86c906956e086c538c5338c766682b088254a0a1c946c365743 WatchSource:0}: Error finding container 16ad3f4a4679f86c906956e086c538c5338c766682b088254a0a1c946c365743: Status 404 returned error can't find the container with id 16ad3f4a4679f86c906956e086c538c5338c766682b088254a0a1c946c365743
Oct 29 22:53:47.730689 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:47.730648    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm" event={"ID":"dacadeaa-93d8-43fe-a074-5d3b4ef8eae6","Type":"ContainerStarted","Data":"2a4bf4188bcea113f9a211be717cfed72e7370b2869e1d758392491f158c2c89"}
Oct 29 22:53:47.730933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:47.730722    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"
Oct 29 22:53:47.730933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:47.730750    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm" event={"ID":"dacadeaa-93d8-43fe-a074-5d3b4ef8eae6","Type":"ContainerStarted","Data":"14910bb220a27678b1148f65b03e429ba9b48f55d0bc8f9507bb430997f23266"}
Oct 29 22:53:47.732522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:47.732492    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5" event={"ID":"89fce30c-5c90-464b-9881-fa411cb2c320","Type":"ContainerStarted","Data":"5ffee7e0b990413ea587c10c2776da9943868422b666118755e3035ff0711e74"}
Oct 29 22:53:47.732640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:47.732541    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5" event={"ID":"89fce30c-5c90-464b-9881-fa411cb2c320","Type":"ContainerStarted","Data":"16ad3f4a4679f86c906956e086c538c5338c766682b088254a0a1c946c365743"}
Oct 29 22:53:47.732640 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:47.732574    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"
Oct 29 22:53:47.774699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:47.774643    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm" podStartSLOduration=1.774629672 podStartE2EDuration="1.774629672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:47.77103883 +0000 UTC m=+2366.682482384" watchObservedRunningTime="2026-10-29 22:53:47.774629672 +0000 UTC m=+2366.686073226"
Oct 29 22:53:47.814387 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:47.814324    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5" podStartSLOduration=1.814305056 podStartE2EDuration="1.814305056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:47.809981121 +0000 UTC m=+2366.721424688" watchObservedRunningTime="2026-10-29 22:53:47.814305056 +0000 UTC m=+2366.725748610"
Oct 29 22:53:53.805589 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:53.805524    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf" podUID="a5788bfa-4108-4ae8-855c-f56ba146d4f0" containerName="main" containerID="cri-o://4a22fade355afaca2ddcca43eb66334c9d339dab6804c64084c65b7535228243" gracePeriod=45
Oct 29 22:53:54.163600 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.163570    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:53:54.259178 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.259139    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-model-cache\") pod \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") "
Oct 29 22:53:54.259364 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.259215    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-dshm\") pod \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") "
Oct 29 22:53:54.259364 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.259299    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5788bfa-4108-4ae8-855c-f56ba146d4f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5788bfa-4108-4ae8-855c-f56ba146d4f0-tls-certs\") pod \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") "
Oct 29 22:53:54.259480 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.259460    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5788bfa-4108-4ae8-855c-f56ba146d4f0-kube-api-access-kzmjz\" (UniqueName: \"kubernetes.io/projected/a5788bfa-4108-4ae8-855c-f56ba146d4f0-kube-api-access-kzmjz\") pod \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") "
Oct 29 22:53:54.259532 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.259483    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-model-cache" pod "a5788bfa-4108-4ae8-855c-f56ba146d4f0" (UID: "a5788bfa-4108-4ae8-855c-f56ba146d4f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:53:54.259575 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.259556    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-home\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-home\") pod \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") "
Oct 29 22:53:54.259625 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.259611    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-tmp-dir\") pod \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\" (UID: \"a5788bfa-4108-4ae8-855c-f56ba146d4f0\") "
Oct 29 22:53:54.260496 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.260466    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-home" pod "a5788bfa-4108-4ae8-855c-f56ba146d4f0" (UID: "a5788bfa-4108-4ae8-855c-f56ba146d4f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:53:54.260681 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.260657    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:53:54.260774 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.260702    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:53:54.261197 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.261136    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-tmp-dir" pod "a5788bfa-4108-4ae8-855c-f56ba146d4f0" (UID: "a5788bfa-4108-4ae8-855c-f56ba146d4f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:53:54.263283 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.263252    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-dshm" pod "a5788bfa-4108-4ae8-855c-f56ba146d4f0" (UID: "a5788bfa-4108-4ae8-855c-f56ba146d4f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:53:54.263389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.263259    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5788bfa-4108-4ae8-855c-f56ba146d4f0-kube-api-access-kzmjz" pod "a5788bfa-4108-4ae8-855c-f56ba146d4f0" (UID: "a5788bfa-4108-4ae8-855c-f56ba146d4f0"). InnerVolumeSpecName "kube-api-access-kzmjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:53:54.264208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.264185    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5788bfa-4108-4ae8-855c-f56ba146d4f0-tls-certs" pod "a5788bfa-4108-4ae8-855c-f56ba146d4f0" (UID: "a5788bfa-4108-4ae8-855c-f56ba146d4f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:53:54.362248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.362144    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:53:54.362248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.362182    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5788bfa-4108-4ae8-855c-f56ba146d4f0-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:53:54.362248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.362195    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5788bfa-4108-4ae8-855c-f56ba146d4f0-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:53:54.362248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.362206    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzmjz\" (UniqueName: \"kubernetes.io/projected/a5788bfa-4108-4ae8-855c-f56ba146d4f0-kube-api-access-kzmjz\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:53:54.794369 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.794333    4559 generic.go:356] "Generic (PLEG): container finished" podID="a5788bfa-4108-4ae8-855c-f56ba146d4f0" containerID="4a22fade355afaca2ddcca43eb66334c9d339dab6804c64084c65b7535228243" exitCode=0
Oct 29 22:53:54.794594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.794390    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"
Oct 29 22:53:54.794594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.794416    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf" event={"ID":"a5788bfa-4108-4ae8-855c-f56ba146d4f0","Type":"ContainerDied","Data":"4a22fade355afaca2ddcca43eb66334c9d339dab6804c64084c65b7535228243"}
Oct 29 22:53:54.794594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.794476    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf" event={"ID":"a5788bfa-4108-4ae8-855c-f56ba146d4f0","Type":"ContainerDied","Data":"94159f17a996d878e1b3fe02a8a0e110e2fde19afb6281c5fe5bcd0d49149e34"}
Oct 29 22:53:54.794594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.794496    4559 scope.go:122] "RemoveContainer" containerID="4a22fade355afaca2ddcca43eb66334c9d339dab6804c64084c65b7535228243"
Oct 29 22:53:54.807292 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.807139    4559 scope.go:122] "RemoveContainer" containerID="4a22fade355afaca2ddcca43eb66334c9d339dab6804c64084c65b7535228243"
Oct 29 22:53:54.807611 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:53:54.807506    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a22fade355afaca2ddcca43eb66334c9d339dab6804c64084c65b7535228243\": container with ID starting with 4a22fade355afaca2ddcca43eb66334c9d339dab6804c64084c65b7535228243 not found: ID does not exist" containerID="4a22fade355afaca2ddcca43eb66334c9d339dab6804c64084c65b7535228243"
Oct 29 22:53:54.807611 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.807547    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a22fade355afaca2ddcca43eb66334c9d339dab6804c64084c65b7535228243"} err="failed to get container status \"4a22fade355afaca2ddcca43eb66334c9d339dab6804c64084c65b7535228243\": rpc error: code = NotFound desc = could not find container \"4a22fade355afaca2ddcca43eb66334c9d339dab6804c64084c65b7535228243\": container with ID starting with 4a22fade355afaca2ddcca43eb66334c9d339dab6804c64084c65b7535228243 not found: ID does not exist"
Oct 29 22:53:54.850085 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.850054    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"]
Oct 29 22:53:54.864321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:54.864285    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-6fc6bd/rate-test-model-6fc6bd-kserve-9df7b6fbf-bkvsf"]
Oct 29 22:53:55.233103 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:55.233067    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5788bfa-4108-4ae8-855c-f56ba146d4f0" path="/var/lib/kubelet/pods/a5788bfa-4108-4ae8-855c-f56ba146d4f0/volumes"
Oct 29 22:53:58.746258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:58.746222    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"
Oct 29 22:53:58.746928 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:53:58.746319    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"
Oct 29 22:54:00.181360 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.181324    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"]
Oct 29 22:54:00.182194 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.182177    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a5788bfa-4108-4ae8-855c-f56ba146d4f0" containerName="main"
Oct 29 22:54:00.182194 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.182193    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5788bfa-4108-4ae8-855c-f56ba146d4f0" containerName="main"
Oct 29 22:54:00.182329 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.182318    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5788bfa-4108-4ae8-855c-f56ba146d4f0" containerName="main"
Oct 29 22:54:00.187832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.187795    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.207415 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.207379    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"]
Oct 29 22:54:00.214710 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.214679    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-af260f-openshift-default-dockercfg-glq4f\""
Oct 29 22:54:00.263154 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.263117    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istiod-ca-cert\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.263369 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.263164    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-token\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.263369 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.263199    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-envoy\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.263369 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.263245    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g942v\" (UniqueName: \"kubernetes.io/projected/f17340ab-f438-4e3f-a5c2-efd46b75b39e-kube-api-access-g942v\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.263369 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.263279    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-data\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.263615 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.263388    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-credential-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.263615 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.263430    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-podinfo\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.263615 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.263470    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-workload-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.263615 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.263566    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-workload-certs\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.365772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.365734    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istiod-ca-cert\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.366037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.365790    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-token\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.366037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.365862    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-envoy\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.366037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.365919    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g942v\" (UniqueName: \"kubernetes.io/projected/f17340ab-f438-4e3f-a5c2-efd46b75b39e-kube-api-access-g942v\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.366037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.365956    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-data\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.366305 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.366045    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-credential-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.366305 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.366083    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-podinfo\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.366305 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.366120    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-workload-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.366305 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.366204    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-workload-certs\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.368699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.366394    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-data\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.368699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.366596    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istiod-ca-cert\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.368699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.366835    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-workload-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.368699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.366892    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-credential-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.368699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.366920    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-workload-certs\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.369397 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.369369    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-envoy\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.369504 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.369377    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-podinfo\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.385394 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.385369    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-token\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.385646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.385626    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g942v\" (UniqueName: \"kubernetes.io/projected/f17340ab-f438-4e3f-a5c2-efd46b75b39e-kube-api-access-g942v\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.502875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.502782    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:00.950673 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:54:00.950635    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17340ab_f438_4e3f_a5c2_efd46b75b39e.slice/crio-306c9174117b96ed7b52007b0e0f522934a987465ffc29e5f18a3d0149c13fad WatchSource:0}: Error finding container 306c9174117b96ed7b52007b0e0f522934a987465ffc29e5f18a3d0149c13fad: Status 404 returned error can't find the container with id 306c9174117b96ed7b52007b0e0f522934a987465ffc29e5f18a3d0149c13fad
Oct 29 22:54:00.954277 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.954233    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:54:00.954386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.954368    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:54:00.954432 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.954417    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:54:00.971524 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.970513    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"]
Oct 29 22:54:00.992192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:00.992153    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v" event={"ID":"f17340ab-f438-4e3f-a5c2-efd46b75b39e","Type":"ContainerStarted","Data":"306c9174117b96ed7b52007b0e0f522934a987465ffc29e5f18a3d0149c13fad"}
Oct 29 22:54:02.002077 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:02.002039    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v" event={"ID":"f17340ab-f438-4e3f-a5c2-efd46b75b39e","Type":"ContainerStarted","Data":"e691707e1d997259ca962c77dae0e1753bfc2bae732069668f88d56b3db258a2"}
Oct 29 22:54:02.504048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:02.503962    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:02.506069 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:02.506039    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 29 22:54:02.506199 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:02.506090    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v" podUID="f17340ab-f438-4e3f-a5c2-efd46b75b39e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 29 22:54:03.503636 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:03.503600    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 29 22:54:03.504097 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:03.503653    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v" podUID="f17340ab-f438-4e3f-a5c2-efd46b75b39e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 29 22:54:04.503723 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:04.503678    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 29 22:54:04.504175 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:04.503732    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v" podUID="f17340ab-f438-4e3f-a5c2-efd46b75b39e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 29 22:54:05.512842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:05.507530    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 29 22:54:05.512842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:05.507598    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v" podUID="f17340ab-f438-4e3f-a5c2-efd46b75b39e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 29 22:54:06.503774 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:06.503730    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 29 22:54:06.504015 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:06.503792    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v" podUID="f17340ab-f438-4e3f-a5c2-efd46b75b39e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 29 22:54:07.503583 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:07.503541    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 29 22:54:07.504020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:07.503619    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v" podUID="f17340ab-f438-4e3f-a5c2-efd46b75b39e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 29 22:54:08.509669 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:08.509634    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:08.510106 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:08.510082    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:08.510908 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:08.510890    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 22:54:08.569647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:08.569576    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v" podStartSLOduration=8.569556294 podStartE2EDuration="8.569556294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:02.069225398 +0000 UTC m=+2380.980668965" watchObservedRunningTime="2026-10-29 22:54:08.569556294 +0000 UTC m=+2387.480999849"
Oct 29 22:54:11.689443 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.689403    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"]
Oct 29 22:54:11.721513 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.721477    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"]
Oct 29 22:54:11.721725 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.721603    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.739907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.739873    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a962d7-openshift-default-dockercfg-mk22d\""
Oct 29 22:54:11.832352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.832311    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-istio-envoy\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.832636 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.832427    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-istio-data\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.832636 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.832456    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-credential-socket\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.832636 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.832487    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08448bc0-6fde-4460-bdf7-d65409faa732-istio-token\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.832636 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.832516    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbplh\" (UniqueName: \"kubernetes.io/projected/08448bc0-6fde-4460-bdf7-d65409faa732-kube-api-access-xbplh\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.832636 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.832607    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-workload-socket\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.832969 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.832643    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08448bc0-6fde-4460-bdf7-d65409faa732-istiod-ca-cert\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.832969 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.832675    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-workload-certs\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.832969 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.832823    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08448bc0-6fde-4460-bdf7-d65409faa732-istio-podinfo\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.934614 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.934579    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08448bc0-6fde-4460-bdf7-d65409faa732-istio-podinfo\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.934864 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.934738    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-istio-envoy\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.934864 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.934851    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-istio-data\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.935001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.934875    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-credential-socket\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.935001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.934894    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08448bc0-6fde-4460-bdf7-d65409faa732-istio-token\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.935001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.934912    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbplh\" (UniqueName: \"kubernetes.io/projected/08448bc0-6fde-4460-bdf7-d65409faa732-kube-api-access-xbplh\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.935119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.935001    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-workload-socket\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.935119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.935041    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08448bc0-6fde-4460-bdf7-d65409faa732-istiod-ca-cert\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.935119 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.935073    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-workload-certs\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.935594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.935559    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-workload-socket\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.935740 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.935577    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-workload-certs\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.935849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.935737    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-istio-data\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.935943 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.935906    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-credential-socket\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.936152 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.936127    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08448bc0-6fde-4460-bdf7-d65409faa732-istiod-ca-cert\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.937646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.937623    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-istio-envoy\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.938188 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.938166    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08448bc0-6fde-4460-bdf7-d65409faa732-istio-podinfo\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.965258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.965184    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08448bc0-6fde-4460-bdf7-d65409faa732-istio-token\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:11.965507 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:11.965480    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbplh\" (UniqueName: \"kubernetes.io/projected/08448bc0-6fde-4460-bdf7-d65409faa732-kube-api-access-xbplh\") pod \"e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") " pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:12.045835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:12.045435    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:12.676731 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:12.676696    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"]
Oct 29 22:54:12.678401 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:54:12.678369    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08448bc0_6fde_4460_bdf7_d65409faa732.slice/crio-057c8bbaf4a3f021e50d5f523fa9a3c2ba70932c186c6f49913f9d8fee139a36 WatchSource:0}: Error finding container 057c8bbaf4a3f021e50d5f523fa9a3c2ba70932c186c6f49913f9d8fee139a36: Status 404 returned error can't find the container with id 057c8bbaf4a3f021e50d5f523fa9a3c2ba70932c186c6f49913f9d8fee139a36
Oct 29 22:54:12.682706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:12.682661    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:54:12.682858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:12.682763    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:54:12.682916 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:12.682887    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:54:13.303794 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:13.303739    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm" event={"ID":"08448bc0-6fde-4460-bdf7-d65409faa732","Type":"ContainerStarted","Data":"8096bfd682074ff9db2696aacd1b98126b100ed8cc05bf544a005060fb106b97"}
Oct 29 22:54:13.304286 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:13.303853    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm" event={"ID":"08448bc0-6fde-4460-bdf7-d65409faa732","Type":"ContainerStarted","Data":"057c8bbaf4a3f021e50d5f523fa9a3c2ba70932c186c6f49913f9d8fee139a36"}
Oct 29 22:54:13.360276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:13.360210    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm" podStartSLOduration=2.360191 podStartE2EDuration="2.360191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:13.355637944 +0000 UTC m=+2392.267081499" watchObservedRunningTime="2026-10-29 22:54:13.360191 +0000 UTC m=+2392.271634554"
Oct 29 22:54:13.892831 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:54:13.887388    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:54:14.054903 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:14.052199    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:14.075711 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:14.068602    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:14.402838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:14.398260    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:14.403533 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:14.403095    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:54:15.257595 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:15.257044    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:15.319001 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:15.318960    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:54:15.401738 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:15.400960    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:15.691933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:15.691884    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:54:15.791920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:15.718848    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:54:15.802059 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:15.718873    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:15.994392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:15.994279    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz" podUID="3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:16.091037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:16.090973    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8" podUID="882dc67e-fe87-4033-8be0-bfdc48845531" containerName="main" probeResult="failure" output="Get \"https://10.134.0.207:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:19.698088 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:19.698049    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"]
Oct 29 22:54:19.703589 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:19.703562    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"
Oct 29 22:54:19.725303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:19.725262    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"]
Oct 29 22:54:19.729583 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:19.729524    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a962d7\""
Oct 29 22:54:19.729583 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:19.729531    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a962d7-dockercfg-kbrss\""
Oct 29 22:54:19.790839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:19.783801    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c569s\" (UniqueName: \"kubernetes.io/projected/4ad36d06-a1b2-4498-8a82-539ff6addb6a-kube-api-access-c569s\") pod \"payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928\" (UID: \"4ad36d06-a1b2-4498-8a82-539ff6addb6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"
Oct 29 22:54:19.790839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:19.784031    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad36d06-a1b2-4498-8a82-539ff6addb6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928\" (UID: \"4ad36d06-a1b2-4498-8a82-539ff6addb6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"
Oct 29 22:54:19.933838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:19.887224    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad36d06-a1b2-4498-8a82-539ff6addb6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928\" (UID: \"4ad36d06-a1b2-4498-8a82-539ff6addb6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"
Oct 29 22:54:19.935252 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:19.935218    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad36d06-a1b2-4498-8a82-539ff6addb6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928\" (UID: \"4ad36d06-a1b2-4498-8a82-539ff6addb6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"
Oct 29 22:54:19.941073 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:19.941038    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c569s\" (UniqueName: \"kubernetes.io/projected/4ad36d06-a1b2-4498-8a82-539ff6addb6a-kube-api-access-c569s\") pod \"payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928\" (UID: \"4ad36d06-a1b2-4498-8a82-539ff6addb6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"
Oct 29 22:54:20.003222 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:19.991895    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c569s\" (UniqueName: \"kubernetes.io/projected/4ad36d06-a1b2-4498-8a82-539ff6addb6a-kube-api-access-c569s\") pod \"payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928\" (UID: \"4ad36d06-a1b2-4498-8a82-539ff6addb6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"
Oct 29 22:54:20.140390 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:20.140346    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"]
Oct 29 22:54:20.189306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:20.189278    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"
Oct 29 22:54:20.211455 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:20.211424    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"]
Oct 29 22:54:20.320569 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:20.320489    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"
Oct 29 22:54:20.364838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:20.359117    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae036a5-0fa1-4aca-8cb8-6b2982c994cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj\" (UID: \"bae036a5-0fa1-4aca-8cb8-6b2982c994cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"
Oct 29 22:54:20.364838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:20.359261    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8p4z\" (UniqueName: \"kubernetes.io/projected/bae036a5-0fa1-4aca-8cb8-6b2982c994cf-kube-api-access-f8p4z\") pod \"payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj\" (UID: \"bae036a5-0fa1-4aca-8cb8-6b2982c994cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"
Oct 29 22:54:20.482223 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:20.482168    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8p4z\" (UniqueName: \"kubernetes.io/projected/bae036a5-0fa1-4aca-8cb8-6b2982c994cf-kube-api-access-f8p4z\") pod \"payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj\" (UID: \"bae036a5-0fa1-4aca-8cb8-6b2982c994cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"
Oct 29 22:54:20.482734 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:20.482640    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae036a5-0fa1-4aca-8cb8-6b2982c994cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj\" (UID: \"bae036a5-0fa1-4aca-8cb8-6b2982c994cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"
Oct 29 22:54:20.484192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:20.484039    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae036a5-0fa1-4aca-8cb8-6b2982c994cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj\" (UID: \"bae036a5-0fa1-4aca-8cb8-6b2982c994cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"
Oct 29 22:54:20.503311 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:20.503279    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8p4z\" (UniqueName: \"kubernetes.io/projected/bae036a5-0fa1-4aca-8cb8-6b2982c994cf-kube-api-access-f8p4z\") pod \"payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj\" (UID: \"bae036a5-0fa1-4aca-8cb8-6b2982c994cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"
Oct 29 22:54:20.517270 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:20.517224    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"
Oct 29 22:54:20.693961 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:20.693767    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"]
Oct 29 22:54:20.694440 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:54:20.694416    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad36d06_a1b2_4498_8a82_539ff6addb6a.slice/crio-d83d693a4101098d9bc5064299b9d73c0e717a023c7f31b603c169f9739335ec WatchSource:0}: Error finding container d83d693a4101098d9bc5064299b9d73c0e717a023c7f31b603c169f9739335ec: Status 404 returned error can't find the container with id d83d693a4101098d9bc5064299b9d73c0e717a023c7f31b603c169f9739335ec
Oct 29 22:54:20.956138 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:20.955980    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"]
Oct 29 22:54:20.980126 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:54:20.980089    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae036a5_0fa1_4aca_8cb8_6b2982c994cf.slice/crio-7389bc03eff4284e9caf4bb9efc0617024b1c57d804790c732a6687fd4f217b2 WatchSource:0}: Error finding container 7389bc03eff4284e9caf4bb9efc0617024b1c57d804790c732a6687fd4f217b2: Status 404 returned error can't find the container with id 7389bc03eff4284e9caf4bb9efc0617024b1c57d804790c732a6687fd4f217b2
Oct 29 22:54:21.625804 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:21.625689    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj" event={"ID":"bae036a5-0fa1-4aca-8cb8-6b2982c994cf","Type":"ContainerStarted","Data":"70cbb3421f08fffb58dcf6f81e2eaef4a8df3d0ad2940817a1e70df27de6493e"}
Oct 29 22:54:21.625804 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:21.625776    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"
Oct 29 22:54:21.625804 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:21.625801    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj" event={"ID":"bae036a5-0fa1-4aca-8cb8-6b2982c994cf","Type":"ContainerStarted","Data":"7389bc03eff4284e9caf4bb9efc0617024b1c57d804790c732a6687fd4f217b2"}
Oct 29 22:54:21.627730 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:21.627699    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928" event={"ID":"4ad36d06-a1b2-4498-8a82-539ff6addb6a","Type":"ContainerStarted","Data":"c372f83448fb50ee70a73b6cc0d5eaf4da1e63f99c21bcc24dda9a1364b87d31"}
Oct 29 22:54:21.627919 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:21.627747    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928" event={"ID":"4ad36d06-a1b2-4498-8a82-539ff6addb6a","Type":"ContainerStarted","Data":"d83d693a4101098d9bc5064299b9d73c0e717a023c7f31b603c169f9739335ec"}
Oct 29 22:54:21.627919 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:21.627789    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"
Oct 29 22:54:21.673573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:21.673500    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj" podStartSLOduration=2.673480604 podStartE2EDuration="2.673480604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:21.663992512 +0000 UTC m=+2400.575436069" watchObservedRunningTime="2026-10-29 22:54:21.673480604 +0000 UTC m=+2400.584924159"
Oct 29 22:54:21.706972 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:21.706912    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928" podStartSLOduration=2.706897896 podStartE2EDuration="2.706897896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:21.702567274 +0000 UTC m=+2400.614010827" watchObservedRunningTime="2026-10-29 22:54:21.706897896 +0000 UTC m=+2400.618341451"
Oct 29 22:54:21.774112 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:21.774076    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:54:21.785631 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:21.785601    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:54:25.063176 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:25.063137    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"]
Oct 29 22:54:25.063720 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:25.063394    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" podUID="b616ef83-09fe-47a2-9d15-4ba5a0126998" containerName="istio-proxy" containerID="cri-o://737a9b1fccf16c1391dd2c2ab48bee5f21be91a8ebdcb68c514b70bc7a3e5eb7" gracePeriod=30
Oct 29 22:54:25.188404 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:25.188364    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"]
Oct 29 22:54:25.188623 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:25.188596    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp" podUID="0eb483d2-a79c-4e78-877e-d2a1f3285613" containerName="istio-proxy" containerID="cri-o://443e78d2c3f97f34d4b0e7c0db4f139cf15d17b89b4e57c92fd519df26e7ecd8" gracePeriod=30
Oct 29 22:54:25.669172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:25.669122    4559 generic.go:356] "Generic (PLEG): container finished" podID="e09d2003-0997-418a-8fa3-f64c92d28572" containerID="01e3d92ce7e5890894e7e1f66c4380832ad287ef5508844f84b3c986e0ca9472" exitCode=0
Oct 29 22:54:25.669172 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:25.669172    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs" event={"ID":"e09d2003-0997-418a-8fa3-f64c92d28572","Type":"ContainerDied","Data":"01e3d92ce7e5890894e7e1f66c4380832ad287ef5508844f84b3c986e0ca9472"}
Oct 29 22:54:25.669895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:25.669801    4559 scope.go:122] "RemoveContainer" containerID="01e3d92ce7e5890894e7e1f66c4380832ad287ef5508844f84b3c986e0ca9472"
Oct 29 22:54:25.672695 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:25.672518    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:54:25.672910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:25.672766    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:54:25.672910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:25.672835    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:54:26.682033 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:26.681996    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs" event={"ID":"e09d2003-0997-418a-8fa3-f64c92d28572","Type":"ContainerStarted","Data":"8397c838f1cde156413c7e241b28be7a820f89ff01bf91ca9d5edf39135866f1"}
Oct 29 22:54:27.506595 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:27.506561    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:54:27.507406 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:27.507088    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 29 22:54:27.507406 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:27.507140    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs" podUID="e09d2003-0997-418a-8fa3-f64c92d28572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 29 22:54:28.502838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:28.500265    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 29 22:54:28.502838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:28.500345    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs" podUID="e09d2003-0997-418a-8fa3-f64c92d28572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 29 22:54:28.583839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:28.581900    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:54:29.500701 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:29.500662    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 29 22:54:29.500942 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:29.500737    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs" podUID="e09d2003-0997-418a-8fa3-f64c92d28572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 29 22:54:30.426047 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.426015    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:54:30.492491 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.492447    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-token\" (UniqueName: \"kubernetes.io/projected/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-token\") pod \"b616ef83-09fe-47a2-9d15-4ba5a0126998\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") "
Oct 29 22:54:30.493009 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.492984    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-workload-certs\") pod \"b616ef83-09fe-47a2-9d15-4ba5a0126998\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") "
Oct 29 22:54:30.493484 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.493453    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-workload-socket\") pod \"b616ef83-09fe-47a2-9d15-4ba5a0126998\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") "
Oct 29 22:54:30.493963 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.493930    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b616ef83-09fe-47a2-9d15-4ba5a0126998-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b616ef83-09fe-47a2-9d15-4ba5a0126998-istiod-ca-cert\") pod \"b616ef83-09fe-47a2-9d15-4ba5a0126998\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") "
Oct 29 22:54:30.495026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.495006    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-data\") pod \"b616ef83-09fe-47a2-9d15-4ba5a0126998\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") "
Oct 29 22:54:30.495262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.495240    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-envoy\") pod \"b616ef83-09fe-47a2-9d15-4ba5a0126998\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") "
Oct 29 22:54:30.495431 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.495414    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-credential-socket\") pod \"b616ef83-09fe-47a2-9d15-4ba5a0126998\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") "
Oct 29 22:54:30.496622 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.496504    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-podinfo\") pod \"b616ef83-09fe-47a2-9d15-4ba5a0126998\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") "
Oct 29 22:54:30.496798 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.496779    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b616ef83-09fe-47a2-9d15-4ba5a0126998-kube-api-access-6rlfk\" (UniqueName: \"kubernetes.io/projected/b616ef83-09fe-47a2-9d15-4ba5a0126998-kube-api-access-6rlfk\") pod \"b616ef83-09fe-47a2-9d15-4ba5a0126998\" (UID: \"b616ef83-09fe-47a2-9d15-4ba5a0126998\") "
Oct 29 22:54:30.507367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.494477    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b616ef83-09fe-47a2-9d15-4ba5a0126998-istiod-ca-cert" pod "b616ef83-09fe-47a2-9d15-4ba5a0126998" (UID: "b616ef83-09fe-47a2-9d15-4ba5a0126998"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:30.507367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.495933    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-credential-socket" pod "b616ef83-09fe-47a2-9d15-4ba5a0126998" (UID: "b616ef83-09fe-47a2-9d15-4ba5a0126998"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:30.507367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.496158    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-data" pod "b616ef83-09fe-47a2-9d15-4ba5a0126998" (UID: "b616ef83-09fe-47a2-9d15-4ba5a0126998"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:30.507367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.503052    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-workload-certs" pod "b616ef83-09fe-47a2-9d15-4ba5a0126998" (UID: "b616ef83-09fe-47a2-9d15-4ba5a0126998"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:30.507367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.503219    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-workload-socket" pod "b616ef83-09fe-47a2-9d15-4ba5a0126998" (UID: "b616ef83-09fe-47a2-9d15-4ba5a0126998"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:30.507367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.503790    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 29 22:54:30.507367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.503913    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs" podUID="e09d2003-0997-418a-8fa3-f64c92d28572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 29 22:54:30.507367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.505413    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.507367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.505436    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.507367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.505454    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.507367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.505471    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.507367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.505486    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b616ef83-09fe-47a2-9d15-4ba5a0126998-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.507367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.507196    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-podinfo" pod "b616ef83-09fe-47a2-9d15-4ba5a0126998" (UID: "b616ef83-09fe-47a2-9d15-4ba5a0126998"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:54:30.507367 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.507330    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b616ef83-09fe-47a2-9d15-4ba5a0126998-kube-api-access-6rlfk" pod "b616ef83-09fe-47a2-9d15-4ba5a0126998" (UID: "b616ef83-09fe-47a2-9d15-4ba5a0126998"). InnerVolumeSpecName "kube-api-access-6rlfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:30.547077 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.529156    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-envoy" pod "b616ef83-09fe-47a2-9d15-4ba5a0126998" (UID: "b616ef83-09fe-47a2-9d15-4ba5a0126998"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:30.617430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.604204    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-token" pod "b616ef83-09fe-47a2-9d15-4ba5a0126998" (UID: "b616ef83-09fe-47a2-9d15-4ba5a0126998"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:30.617430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.613026    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.617430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.613062    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.617430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.613077    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rlfk\" (UniqueName: \"kubernetes.io/projected/b616ef83-09fe-47a2-9d15-4ba5a0126998-kube-api-access-6rlfk\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.617430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.613089    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b616ef83-09fe-47a2-9d15-4ba5a0126998-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.694939 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.694913    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:54:30.713626 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.713596    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-workload-socket\") pod \"0eb483d2-a79c-4e78-877e-d2a1f3285613\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") "
Oct 29 22:54:30.713786 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.713656    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-envoy\") pod \"0eb483d2-a79c-4e78-877e-d2a1f3285613\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") "
Oct 29 22:54:30.713786 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.713683    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-data\") pod \"0eb483d2-a79c-4e78-877e-d2a1f3285613\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") "
Oct 29 22:54:30.713786 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.713717    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-workload-certs\") pod \"0eb483d2-a79c-4e78-877e-d2a1f3285613\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") "
Oct 29 22:54:30.713786 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.713778    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-token\" (UniqueName: \"kubernetes.io/projected/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-token\") pod \"0eb483d2-a79c-4e78-877e-d2a1f3285613\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") "
Oct 29 22:54:30.714064 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.713907    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb483d2-a79c-4e78-877e-d2a1f3285613-kube-api-access-ggw4p\" (UniqueName: \"kubernetes.io/projected/0eb483d2-a79c-4e78-877e-d2a1f3285613-kube-api-access-ggw4p\") pod \"0eb483d2-a79c-4e78-877e-d2a1f3285613\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") "
Oct 29 22:54:30.714064 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.713928    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-workload-socket" pod "0eb483d2-a79c-4e78-877e-d2a1f3285613" (UID: "0eb483d2-a79c-4e78-877e-d2a1f3285613"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:30.714064 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.713983    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb483d2-a79c-4e78-877e-d2a1f3285613-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eb483d2-a79c-4e78-877e-d2a1f3285613-istiod-ca-cert\") pod \"0eb483d2-a79c-4e78-877e-d2a1f3285613\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") "
Oct 29 22:54:30.714064 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.714026    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-podinfo\") pod \"0eb483d2-a79c-4e78-877e-d2a1f3285613\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") "
Oct 29 22:54:30.714064 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.714060    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-credential-socket\") pod \"0eb483d2-a79c-4e78-877e-d2a1f3285613\" (UID: \"0eb483d2-a79c-4e78-877e-d2a1f3285613\") "
Oct 29 22:54:30.714322 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.714211    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-data" pod "0eb483d2-a79c-4e78-877e-d2a1f3285613" (UID: "0eb483d2-a79c-4e78-877e-d2a1f3285613"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:30.714364 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.714346    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb483d2-a79c-4e78-877e-d2a1f3285613-istiod-ca-cert" pod "0eb483d2-a79c-4e78-877e-d2a1f3285613" (UID: "0eb483d2-a79c-4e78-877e-d2a1f3285613"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:30.714669 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.714540    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-credential-socket" pod "0eb483d2-a79c-4e78-877e-d2a1f3285613" (UID: "0eb483d2-a79c-4e78-877e-d2a1f3285613"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:30.714669 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.714556    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-workload-certs" pod "0eb483d2-a79c-4e78-877e-d2a1f3285613" (UID: "0eb483d2-a79c-4e78-877e-d2a1f3285613"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:30.715247 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.715221    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eb483d2-a79c-4e78-877e-d2a1f3285613-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.715326 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.715253    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.715326 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.715268    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.715326 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.715280    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.715326 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.715292    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.716198 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.716167    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-envoy" pod "0eb483d2-a79c-4e78-877e-d2a1f3285613" (UID: "0eb483d2-a79c-4e78-877e-d2a1f3285613"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:54:30.716409 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.716388    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb483d2-a79c-4e78-877e-d2a1f3285613-kube-api-access-ggw4p" pod "0eb483d2-a79c-4e78-877e-d2a1f3285613" (UID: "0eb483d2-a79c-4e78-877e-d2a1f3285613"). InnerVolumeSpecName "kube-api-access-ggw4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:30.716457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.716410    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-token" pod "0eb483d2-a79c-4e78-877e-d2a1f3285613" (UID: "0eb483d2-a79c-4e78-877e-d2a1f3285613"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:30.716511 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.716472    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-podinfo" pod "0eb483d2-a79c-4e78-877e-d2a1f3285613" (UID: "0eb483d2-a79c-4e78-877e-d2a1f3285613"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:54:30.774271 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.772584    4559 generic.go:356] "Generic (PLEG): container finished" podID="b616ef83-09fe-47a2-9d15-4ba5a0126998" containerID="737a9b1fccf16c1391dd2c2ab48bee5f21be91a8ebdcb68c514b70bc7a3e5eb7" exitCode=0
Oct 29 22:54:30.774271 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.772941    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"
Oct 29 22:54:30.774271 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.773901    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" event={"ID":"b616ef83-09fe-47a2-9d15-4ba5a0126998","Type":"ContainerDied","Data":"737a9b1fccf16c1391dd2c2ab48bee5f21be91a8ebdcb68c514b70bc7a3e5eb7"}
Oct 29 22:54:30.774271 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.773981    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7" event={"ID":"b616ef83-09fe-47a2-9d15-4ba5a0126998","Type":"ContainerDied","Data":"ba5501565a043825784151731b175b244f2a96bff1a4696eb8355ff59bbedf2e"}
Oct 29 22:54:30.774271 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.774010    4559 scope.go:122] "RemoveContainer" containerID="737a9b1fccf16c1391dd2c2ab48bee5f21be91a8ebdcb68c514b70bc7a3e5eb7"
Oct 29 22:54:30.778594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.776128    4559 generic.go:356] "Generic (PLEG): container finished" podID="0eb483d2-a79c-4e78-877e-d2a1f3285613" containerID="443e78d2c3f97f34d4b0e7c0db4f139cf15d17b89b4e57c92fd519df26e7ecd8" exitCode=0
Oct 29 22:54:30.778594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.776177    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp" event={"ID":"0eb483d2-a79c-4e78-877e-d2a1f3285613","Type":"ContainerDied","Data":"443e78d2c3f97f34d4b0e7c0db4f139cf15d17b89b4e57c92fd519df26e7ecd8"}
Oct 29 22:54:30.778594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.776209    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp" event={"ID":"0eb483d2-a79c-4e78-877e-d2a1f3285613","Type":"ContainerDied","Data":"2944595e062cfc7e9c7552e9b94d69b20e7ecf374cac5685ddcf467d35fa3d53"}
Oct 29 22:54:30.778594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.776267    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"
Oct 29 22:54:30.793873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.793652    4559 scope.go:122] "RemoveContainer" containerID="737a9b1fccf16c1391dd2c2ab48bee5f21be91a8ebdcb68c514b70bc7a3e5eb7"
Oct 29 22:54:30.794176 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:54:30.794153    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"737a9b1fccf16c1391dd2c2ab48bee5f21be91a8ebdcb68c514b70bc7a3e5eb7\": container with ID starting with 737a9b1fccf16c1391dd2c2ab48bee5f21be91a8ebdcb68c514b70bc7a3e5eb7 not found: ID does not exist" containerID="737a9b1fccf16c1391dd2c2ab48bee5f21be91a8ebdcb68c514b70bc7a3e5eb7"
Oct 29 22:54:30.794278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.794186    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737a9b1fccf16c1391dd2c2ab48bee5f21be91a8ebdcb68c514b70bc7a3e5eb7"} err="failed to get container status \"737a9b1fccf16c1391dd2c2ab48bee5f21be91a8ebdcb68c514b70bc7a3e5eb7\": rpc error: code = NotFound desc = could not find container \"737a9b1fccf16c1391dd2c2ab48bee5f21be91a8ebdcb68c514b70bc7a3e5eb7\": container with ID starting with 737a9b1fccf16c1391dd2c2ab48bee5f21be91a8ebdcb68c514b70bc7a3e5eb7 not found: ID does not exist"
Oct 29 22:54:30.794278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.794204    4559 scope.go:122] "RemoveContainer" containerID="443e78d2c3f97f34d4b0e7c0db4f139cf15d17b89b4e57c92fd519df26e7ecd8"
Oct 29 22:54:30.810979 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.810860    4559 scope.go:122] "RemoveContainer" containerID="b2c6caf9deb461614ae4687c49233327123a59e6c6a26fa1c689bd1410b6cb9e"
Oct 29 22:54:30.816558 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.816526    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggw4p\" (UniqueName: \"kubernetes.io/projected/0eb483d2-a79c-4e78-877e-d2a1f3285613-kube-api-access-ggw4p\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.816558 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.816564    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.816767 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.816580    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.816767 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.816592    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eb483d2-a79c-4e78-877e-d2a1f3285613-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:54:30.822167 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.822142    4559 scope.go:122] "RemoveContainer" containerID="443e78d2c3f97f34d4b0e7c0db4f139cf15d17b89b4e57c92fd519df26e7ecd8"
Oct 29 22:54:30.822526 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:54:30.822500    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443e78d2c3f97f34d4b0e7c0db4f139cf15d17b89b4e57c92fd519df26e7ecd8\": container with ID starting with 443e78d2c3f97f34d4b0e7c0db4f139cf15d17b89b4e57c92fd519df26e7ecd8 not found: ID does not exist" containerID="443e78d2c3f97f34d4b0e7c0db4f139cf15d17b89b4e57c92fd519df26e7ecd8"
Oct 29 22:54:30.822626 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.822536    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443e78d2c3f97f34d4b0e7c0db4f139cf15d17b89b4e57c92fd519df26e7ecd8"} err="failed to get container status \"443e78d2c3f97f34d4b0e7c0db4f139cf15d17b89b4e57c92fd519df26e7ecd8\": rpc error: code = NotFound desc = could not find container \"443e78d2c3f97f34d4b0e7c0db4f139cf15d17b89b4e57c92fd519df26e7ecd8\": container with ID starting with 443e78d2c3f97f34d4b0e7c0db4f139cf15d17b89b4e57c92fd519df26e7ecd8 not found: ID does not exist"
Oct 29 22:54:30.822626 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.822559    4559 scope.go:122] "RemoveContainer" containerID="b2c6caf9deb461614ae4687c49233327123a59e6c6a26fa1c689bd1410b6cb9e"
Oct 29 22:54:30.822887 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:54:30.822860    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c6caf9deb461614ae4687c49233327123a59e6c6a26fa1c689bd1410b6cb9e\": container with ID starting with b2c6caf9deb461614ae4687c49233327123a59e6c6a26fa1c689bd1410b6cb9e not found: ID does not exist" containerID="b2c6caf9deb461614ae4687c49233327123a59e6c6a26fa1c689bd1410b6cb9e"
Oct 29 22:54:30.822985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.822898    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c6caf9deb461614ae4687c49233327123a59e6c6a26fa1c689bd1410b6cb9e"} err="failed to get container status \"b2c6caf9deb461614ae4687c49233327123a59e6c6a26fa1c689bd1410b6cb9e\": rpc error: code = NotFound desc = could not find container \"b2c6caf9deb461614ae4687c49233327123a59e6c6a26fa1c689bd1410b6cb9e\": container with ID starting with b2c6caf9deb461614ae4687c49233327123a59e6c6a26fa1c689bd1410b6cb9e not found: ID does not exist"
Oct 29 22:54:30.888320 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.888243    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"]
Oct 29 22:54:30.892422 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.892392    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-448t7"]
Oct 29 22:54:30.913994 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.913963    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"]
Oct 29 22:54:30.922743 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:30.922704    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-hz9sp"]
Oct 29 22:54:31.231254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:31.231211    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb483d2-a79c-4e78-877e-d2a1f3285613" path="/var/lib/kubelet/pods/0eb483d2-a79c-4e78-877e-d2a1f3285613/volumes"
Oct 29 22:54:31.231906 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:31.231883    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b616ef83-09fe-47a2-9d15-4ba5a0126998" path="/var/lib/kubelet/pods/b616ef83-09fe-47a2-9d15-4ba5a0126998/volumes"
Oct 29 22:54:31.499980 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:31.499936    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 29 22:54:31.500513 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:31.500013    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs" podUID="e09d2003-0997-418a-8fa3-f64c92d28572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 29 22:54:32.640889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:32.640860    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"
Oct 29 22:54:32.641372 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:32.640934    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"
Oct 29 22:54:32.653909 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:32.653864    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:54:32.655628 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:32.655601    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 22:54:43.847319 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:54:43.847263    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:54:44.423284 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.423252    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"]
Oct 29 22:54:44.424232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.424204    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b616ef83-09fe-47a2-9d15-4ba5a0126998" containerName="istio-proxy"
Oct 29 22:54:44.424232 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.424229    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b616ef83-09fe-47a2-9d15-4ba5a0126998" containerName="istio-proxy"
Oct 29 22:54:44.424412 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.424294    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0eb483d2-a79c-4e78-877e-d2a1f3285613" containerName="istio-proxy"
Oct 29 22:54:44.424412 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.424303    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb483d2-a79c-4e78-877e-d2a1f3285613" containerName="istio-proxy"
Oct 29 22:54:44.424412 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.424338    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0eb483d2-a79c-4e78-877e-d2a1f3285613" containerName="istio-proxy"
Oct 29 22:54:44.424412 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.424346    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb483d2-a79c-4e78-877e-d2a1f3285613" containerName="istio-proxy"
Oct 29 22:54:44.424598 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.424518    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb483d2-a79c-4e78-877e-d2a1f3285613" containerName="istio-proxy"
Oct 29 22:54:44.424598 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.424532    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb483d2-a79c-4e78-877e-d2a1f3285613" containerName="istio-proxy"
Oct 29 22:54:44.424598 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.424549    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b616ef83-09fe-47a2-9d15-4ba5a0126998" containerName="istio-proxy"
Oct 29 22:54:44.431049 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.431026    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.453137 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.453105    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"]
Oct 29 22:54:44.515652 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.515616    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-istio-envoy\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.515844 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.515675    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-credential-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.515896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.515863    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1423563c-a109-46a3-9fd5-cccab25939d3-istio-podinfo\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.516082 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.516063    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-istio-data\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.516125 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.516102    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1423563c-a109-46a3-9fd5-cccab25939d3-istiod-ca-cert\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.516173 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.516127    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-workload-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.516229 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.516164    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1423563c-a109-46a3-9fd5-cccab25939d3-istio-token\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.516274 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.516261    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4gpp\" (UniqueName: \"kubernetes.io/projected/1423563c-a109-46a3-9fd5-cccab25939d3-kube-api-access-k4gpp\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.516320 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.516290    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-workload-certs\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.618208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.618159    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-istio-data\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.618208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.618210    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1423563c-a109-46a3-9fd5-cccab25939d3-istiod-ca-cert\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.618480 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.618231    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-workload-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.618480 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.618318    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1423563c-a109-46a3-9fd5-cccab25939d3-istio-token\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.618480 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.618383    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4gpp\" (UniqueName: \"kubernetes.io/projected/1423563c-a109-46a3-9fd5-cccab25939d3-kube-api-access-k4gpp\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.618480 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.618429    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-workload-certs\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.618785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.618549    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-istio-envoy\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.618785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.618595    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-credential-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.618785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.618600    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-workload-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.618785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.618724    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1423563c-a109-46a3-9fd5-cccab25939d3-istio-podinfo\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.619036 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.618856    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-workload-certs\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.619036 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.618918    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-credential-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.619152 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.619067    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-istio-data\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.619152 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.619076    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1423563c-a109-46a3-9fd5-cccab25939d3-istiod-ca-cert\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.621115 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.621086    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-istio-envoy\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.621369 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.621351    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1423563c-a109-46a3-9fd5-cccab25939d3-istio-podinfo\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.640298 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.640266    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1423563c-a109-46a3-9fd5-cccab25939d3-istio-token\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.640464 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.640439    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4gpp\" (UniqueName: \"kubernetes.io/projected/1423563c-a109-46a3-9fd5-cccab25939d3-kube-api-access-k4gpp\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:44.747657 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:44.747564    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:45.078351 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:45.078319    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"]
Oct 29 22:54:45.079147 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:54:45.079123    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1423563c_a109_46a3_9fd5_cccab25939d3.slice/crio-6e5abe777d0639cfba67b459d0eee9e7f743e2dfe1e9ce42d4740644fd4e6f8b WatchSource:0}: Error finding container 6e5abe777d0639cfba67b459d0eee9e7f743e2dfe1e9ce42d4740644fd4e6f8b: Status 404 returned error can't find the container with id 6e5abe777d0639cfba67b459d0eee9e7f743e2dfe1e9ce42d4740644fd4e6f8b
Oct 29 22:54:45.088553 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:45.088510    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:54:45.088671 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:45.088606    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:54:45.088671 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:45.088656    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:54:45.975353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:45.975316    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958" event={"ID":"1423563c-a109-46a3-9fd5-cccab25939d3","Type":"ContainerStarted","Data":"23184661f506410278cae1b4a7f9a1073c924b89f83535a864bf92a80157285e"}
Oct 29 22:54:45.975550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:45.975371    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958" event={"ID":"1423563c-a109-46a3-9fd5-cccab25939d3","Type":"ContainerStarted","Data":"6e5abe777d0639cfba67b459d0eee9e7f743e2dfe1e9ce42d4740644fd4e6f8b"}
Oct 29 22:54:46.068834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:46.053518    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958" podStartSLOduration=2.053499057 podStartE2EDuration="2.053499057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:46.02921534 +0000 UTC m=+2424.940658893" watchObservedRunningTime="2026-10-29 22:54:46.053499057 +0000 UTC m=+2424.964942613"
Oct 29 22:54:46.750837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:46.750231    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:46.761850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:46.760796    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 29 22:54:46.761850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:46.760877    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958" podUID="1423563c-a109-46a3-9fd5-cccab25939d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 29 22:54:47.748837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:47.748263    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 29 22:54:47.748837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:47.748332    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958" podUID="1423563c-a109-46a3-9fd5-cccab25939d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 29 22:54:48.748422 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:48.748383    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 29 22:54:48.749075 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:48.749043    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958" podUID="1423563c-a109-46a3-9fd5-cccab25939d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 29 22:54:49.760838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:49.750214    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 29 22:54:49.760838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:49.750279    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958" podUID="1423563c-a109-46a3-9fd5-cccab25939d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 29 22:54:50.751166 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:50.751129    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 29 22:54:50.751496 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:50.751454    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958" podUID="1423563c-a109-46a3-9fd5-cccab25939d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 29 22:54:51.909150 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:51.909092    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:51.910227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:51.910065    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:51.910923 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:51.910899    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 22:54:57.968157 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:57.968119    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"]
Oct 29 22:54:57.973794 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:57.973772    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:54:58.002239 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:58.002205    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8c4b02-serving-cert\""
Oct 29 22:54:58.010973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:58.010936    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"]
Oct 29 22:54:58.046255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:58.044831    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c86b737-9464-4a60-9934-64096680f5f3-metrics-certs\") pod \"maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm\" (UID: \"1c86b737-9464-4a60-9934-64096680f5f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:54:58.046255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:58.045083    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c86b737-9464-4a60-9934-64096680f5f3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm\" (UID: \"1c86b737-9464-4a60-9934-64096680f5f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:54:58.046255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:58.045113    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7fjk\" (UniqueName: \"kubernetes.io/projected/1c86b737-9464-4a60-9934-64096680f5f3-kube-api-access-w7fjk\") pod \"maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm\" (UID: \"1c86b737-9464-4a60-9934-64096680f5f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:54:58.046255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:58.045146    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c86b737-9464-4a60-9934-64096680f5f3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm\" (UID: \"1c86b737-9464-4a60-9934-64096680f5f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:54:58.146477 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:58.146442    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c86b737-9464-4a60-9934-64096680f5f3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm\" (UID: \"1c86b737-9464-4a60-9934-64096680f5f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:54:58.146695 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:58.146487    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7fjk\" (UniqueName: \"kubernetes.io/projected/1c86b737-9464-4a60-9934-64096680f5f3-kube-api-access-w7fjk\") pod \"maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm\" (UID: \"1c86b737-9464-4a60-9934-64096680f5f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:54:58.146695 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:58.146515    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c86b737-9464-4a60-9934-64096680f5f3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm\" (UID: \"1c86b737-9464-4a60-9934-64096680f5f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:54:58.146695 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:58.146666    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c86b737-9464-4a60-9934-64096680f5f3-metrics-certs\") pod \"maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm\" (UID: \"1c86b737-9464-4a60-9934-64096680f5f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:54:58.147297 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:58.147275    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c86b737-9464-4a60-9934-64096680f5f3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm\" (UID: \"1c86b737-9464-4a60-9934-64096680f5f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:54:58.149424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:58.149391    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c86b737-9464-4a60-9934-64096680f5f3-metrics-certs\") pod \"maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm\" (UID: \"1c86b737-9464-4a60-9934-64096680f5f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:54:58.149681 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:58.149661    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c86b737-9464-4a60-9934-64096680f5f3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm\" (UID: \"1c86b737-9464-4a60-9934-64096680f5f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:54:58.199847 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:58.193508    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7fjk\" (UniqueName: \"kubernetes.io/projected/1c86b737-9464-4a60-9934-64096680f5f3-kube-api-access-w7fjk\") pod \"maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm\" (UID: \"1c86b737-9464-4a60-9934-64096680f5f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:54:58.294120 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:58.294022    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:54:58.619052 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:54:58.619019    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c86b737_9464_4a60_9934_64096680f5f3.slice/crio-898a1af21f3fffb2182c2e07b4d2a0a235b6645d5b9c0487ba6cf8a99bfc8d88 WatchSource:0}: Error finding container 898a1af21f3fffb2182c2e07b4d2a0a235b6645d5b9c0487ba6cf8a99bfc8d88: Status 404 returned error can't find the container with id 898a1af21f3fffb2182c2e07b4d2a0a235b6645d5b9c0487ba6cf8a99bfc8d88
Oct 29 22:54:58.619742 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:58.619717    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"]
Oct 29 22:54:58.621168 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:58.621146    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:54:59.131659 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:59.131611    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm" event={"ID":"1c86b737-9464-4a60-9934-64096680f5f3","Type":"ContainerStarted","Data":"2f95d835c008bc3bd230bd6b0be4045c5f641409f2d731ec368d753b7c32d858"}
Oct 29 22:54:59.132158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:59.131835    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:54:59.132158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:59.131867    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm" event={"ID":"1c86b737-9464-4a60-9934-64096680f5f3","Type":"ContainerStarted","Data":"898a1af21f3fffb2182c2e07b4d2a0a235b6645d5b9c0487ba6cf8a99bfc8d88"}
Oct 29 22:54:59.176051 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:54:59.175928    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm" podStartSLOduration=1.8798008419999999 podStartE2EDuration="2.175907951s" totalImagesPullingTime="296.107109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:59.171547579 +0000 UTC m=+2438.082991135" watchObservedRunningTime="2026-10-29 22:54:59.175907951 +0000 UTC m=+2438.087351505"
Oct 29 22:55:02.553837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:02.552424    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:55:05.184839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:05.177130    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:55:05.646019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:05.645068    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:05.690993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:05.679955    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:06.079895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:06.072949    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8" podUID="882dc67e-fe87-4033-8be0-bfdc48845531" containerName="main" probeResult="failure" output="Get \"https://10.134.0.207:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:13.695974 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:55:13.695937    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:55:16.925556 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:16.925519    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"]
Oct 29 22:55:16.957022 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:16.956985    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"]
Oct 29 22:55:21.927471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:21.927419    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5" podUID="89fce30c-5c90-464b-9881-fa411cb2c320" containerName="payload-processing" containerID="cri-o://5ffee7e0b990413ea587c10c2776da9943868422b666118755e3035ff0711e74" gracePeriod=25
Oct 29 22:55:21.957906 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:21.957864    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm" podUID="dacadeaa-93d8-43fe-a074-5d3b4ef8eae6" containerName="payload-pre-processing" containerID="cri-o://2a4bf4188bcea113f9a211be717cfed72e7370b2869e1d758392491f158c2c89" gracePeriod=25
Oct 29 22:55:22.262376 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.262348    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"
Oct 29 22:55:22.310169 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.307679    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"
Oct 29 22:55:22.433783 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.433701    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dacadeaa-93d8-43fe-a074-5d3b4ef8eae6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacadeaa-93d8-43fe-a074-5d3b4ef8eae6-plugins-config-volume\") pod \"dacadeaa-93d8-43fe-a074-5d3b4ef8eae6\" (UID: \"dacadeaa-93d8-43fe-a074-5d3b4ef8eae6\") "
Oct 29 22:55:22.433957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.433858    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dacadeaa-93d8-43fe-a074-5d3b4ef8eae6-kube-api-access-6dnvg\" (UniqueName: \"kubernetes.io/projected/dacadeaa-93d8-43fe-a074-5d3b4ef8eae6-kube-api-access-6dnvg\") pod \"dacadeaa-93d8-43fe-a074-5d3b4ef8eae6\" (UID: \"dacadeaa-93d8-43fe-a074-5d3b4ef8eae6\") "
Oct 29 22:55:22.433957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.433879    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89fce30c-5c90-464b-9881-fa411cb2c320-kube-api-access-qv5q4\" (UniqueName: \"kubernetes.io/projected/89fce30c-5c90-464b-9881-fa411cb2c320-kube-api-access-qv5q4\") pod \"89fce30c-5c90-464b-9881-fa411cb2c320\" (UID: \"89fce30c-5c90-464b-9881-fa411cb2c320\") "
Oct 29 22:55:22.433957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.433926    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89fce30c-5c90-464b-9881-fa411cb2c320-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fce30c-5c90-464b-9881-fa411cb2c320-plugins-config-volume\") pod \"89fce30c-5c90-464b-9881-fa411cb2c320\" (UID: \"89fce30c-5c90-464b-9881-fa411cb2c320\") "
Oct 29 22:55:22.434317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.434293    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dacadeaa-93d8-43fe-a074-5d3b4ef8eae6-plugins-config-volume" pod "dacadeaa-93d8-43fe-a074-5d3b4ef8eae6" (UID: "dacadeaa-93d8-43fe-a074-5d3b4ef8eae6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:22.434435 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.434354    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89fce30c-5c90-464b-9881-fa411cb2c320-plugins-config-volume" pod "89fce30c-5c90-464b-9881-fa411cb2c320" (UID: "89fce30c-5c90-464b-9881-fa411cb2c320"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:22.434778 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.434761    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fce30c-5c90-464b-9881-fa411cb2c320-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:55:22.434861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.434781    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacadeaa-93d8-43fe-a074-5d3b4ef8eae6-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:55:22.436211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.436184    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dacadeaa-93d8-43fe-a074-5d3b4ef8eae6-kube-api-access-6dnvg" pod "dacadeaa-93d8-43fe-a074-5d3b4ef8eae6" (UID: "dacadeaa-93d8-43fe-a074-5d3b4ef8eae6"). InnerVolumeSpecName "kube-api-access-6dnvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:22.436314 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.436215    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89fce30c-5c90-464b-9881-fa411cb2c320-kube-api-access-qv5q4" pod "89fce30c-5c90-464b-9881-fa411cb2c320" (UID: "89fce30c-5c90-464b-9881-fa411cb2c320"). InnerVolumeSpecName "kube-api-access-qv5q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:22.468512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.468476    4559 generic.go:356] "Generic (PLEG): container finished" podID="dacadeaa-93d8-43fe-a074-5d3b4ef8eae6" containerID="2a4bf4188bcea113f9a211be717cfed72e7370b2869e1d758392491f158c2c89" exitCode=0
Oct 29 22:55:22.468512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.468524    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"
Oct 29 22:55:22.468836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.468553    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm" event={"ID":"dacadeaa-93d8-43fe-a074-5d3b4ef8eae6","Type":"ContainerDied","Data":"2a4bf4188bcea113f9a211be717cfed72e7370b2869e1d758392491f158c2c89"}
Oct 29 22:55:22.468836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.468621    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm" event={"ID":"dacadeaa-93d8-43fe-a074-5d3b4ef8eae6","Type":"ContainerDied","Data":"14910bb220a27678b1148f65b03e429ba9b48f55d0bc8f9507bb430997f23266"}
Oct 29 22:55:22.468836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.468651    4559 scope.go:122] "RemoveContainer" containerID="2a4bf4188bcea113f9a211be717cfed72e7370b2869e1d758392491f158c2c89"
Oct 29 22:55:22.470635 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.470609    4559 generic.go:356] "Generic (PLEG): container finished" podID="89fce30c-5c90-464b-9881-fa411cb2c320" containerID="5ffee7e0b990413ea587c10c2776da9943868422b666118755e3035ff0711e74" exitCode=0
Oct 29 22:55:22.470779 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.470654    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"
Oct 29 22:55:22.470779 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.470682    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5" event={"ID":"89fce30c-5c90-464b-9881-fa411cb2c320","Type":"ContainerDied","Data":"5ffee7e0b990413ea587c10c2776da9943868422b666118755e3035ff0711e74"}
Oct 29 22:55:22.470779 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.470724    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5" event={"ID":"89fce30c-5c90-464b-9881-fa411cb2c320","Type":"ContainerDied","Data":"16ad3f4a4679f86c906956e086c538c5338c766682b088254a0a1c946c365743"}
Oct 29 22:55:22.481323 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.481298    4559 scope.go:122] "RemoveContainer" containerID="2a4bf4188bcea113f9a211be717cfed72e7370b2869e1d758392491f158c2c89"
Oct 29 22:55:22.481628 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:55:22.481602    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4bf4188bcea113f9a211be717cfed72e7370b2869e1d758392491f158c2c89\": container with ID starting with 2a4bf4188bcea113f9a211be717cfed72e7370b2869e1d758392491f158c2c89 not found: ID does not exist" containerID="2a4bf4188bcea113f9a211be717cfed72e7370b2869e1d758392491f158c2c89"
Oct 29 22:55:22.481716 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.481638    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4bf4188bcea113f9a211be717cfed72e7370b2869e1d758392491f158c2c89"} err="failed to get container status \"2a4bf4188bcea113f9a211be717cfed72e7370b2869e1d758392491f158c2c89\": rpc error: code = NotFound desc = could not find container \"2a4bf4188bcea113f9a211be717cfed72e7370b2869e1d758392491f158c2c89\": container with ID starting with 2a4bf4188bcea113f9a211be717cfed72e7370b2869e1d758392491f158c2c89 not found: ID does not exist"
Oct 29 22:55:22.481716 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.481655    4559 scope.go:122] "RemoveContainer" containerID="5ffee7e0b990413ea587c10c2776da9943868422b666118755e3035ff0711e74"
Oct 29 22:55:22.492390 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.492372    4559 scope.go:122] "RemoveContainer" containerID="5ffee7e0b990413ea587c10c2776da9943868422b666118755e3035ff0711e74"
Oct 29 22:55:22.492709 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:55:22.492680    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ffee7e0b990413ea587c10c2776da9943868422b666118755e3035ff0711e74\": container with ID starting with 5ffee7e0b990413ea587c10c2776da9943868422b666118755e3035ff0711e74 not found: ID does not exist" containerID="5ffee7e0b990413ea587c10c2776da9943868422b666118755e3035ff0711e74"
Oct 29 22:55:22.492785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.492721    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ffee7e0b990413ea587c10c2776da9943868422b666118755e3035ff0711e74"} err="failed to get container status \"5ffee7e0b990413ea587c10c2776da9943868422b666118755e3035ff0711e74\": rpc error: code = NotFound desc = could not find container \"5ffee7e0b990413ea587c10c2776da9943868422b666118755e3035ff0711e74\": container with ID starting with 5ffee7e0b990413ea587c10c2776da9943868422b666118755e3035ff0711e74 not found: ID does not exist"
Oct 29 22:55:22.522681 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.522644    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"]
Oct 29 22:55:22.527146 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.527123    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7be71d-59d98d4c64-5gtp5"]
Oct 29 22:55:22.536869 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.536832    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dnvg\" (UniqueName: \"kubernetes.io/projected/dacadeaa-93d8-43fe-a074-5d3b4ef8eae6-kube-api-access-6dnvg\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:55:22.536869 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.536867    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv5q4\" (UniqueName: \"kubernetes.io/projected/89fce30c-5c90-464b-9881-fa411cb2c320-kube-api-access-qv5q4\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:55:22.561796 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.561758    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"]
Oct 29 22:55:22.566330 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:22.566300    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-648bf674d7-kqdsm"]
Oct 29 22:55:23.231985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:23.231890    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89fce30c-5c90-464b-9881-fa411cb2c320" path="/var/lib/kubelet/pods/89fce30c-5c90-464b-9881-fa411cb2c320/volumes"
Oct 29 22:55:23.232925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:23.232901    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dacadeaa-93d8-43fe-a074-5d3b4ef8eae6" path="/var/lib/kubelet/pods/dacadeaa-93d8-43fe-a074-5d3b4ef8eae6/volumes"
Oct 29 22:55:24.394377 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:24.394341    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"]
Oct 29 22:55:24.394889 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:24.394553    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks" podUID="7f3dffab-440d-4fb7-81bb-9c63ed54d908" containerName="istio-proxy" containerID="cri-o://ffc4eb0a936fcbd2f7d4753d6a50390117a552d7cf382444b7ad6bab7125ced2" gracePeriod=30
Oct 29 22:55:25.142898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.142861    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"]
Oct 29 22:55:25.144005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.143977    4559 cpu_manager.go:414] "containerMap: removing container" podUID="89fce30c-5c90-464b-9881-fa411cb2c320" containerName="payload-processing"
Oct 29 22:55:25.144005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.144003    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89fce30c-5c90-464b-9881-fa411cb2c320" containerName="payload-processing"
Oct 29 22:55:25.144240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.144057    4559 cpu_manager.go:414] "containerMap: removing container" podUID="dacadeaa-93d8-43fe-a074-5d3b4ef8eae6" containerName="payload-pre-processing"
Oct 29 22:55:25.144240 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.144065    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dacadeaa-93d8-43fe-a074-5d3b4ef8eae6" containerName="payload-pre-processing"
Oct 29 22:55:25.144351 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.144243    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="dacadeaa-93d8-43fe-a074-5d3b4ef8eae6" containerName="payload-pre-processing"
Oct 29 22:55:25.144351 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.144271    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="89fce30c-5c90-464b-9881-fa411cb2c320" containerName="payload-processing"
Oct 29 22:55:25.150457 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.150433    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.169734 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.169705    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"]
Oct 29 22:55:25.176415 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.176387    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-3ee017-openshift-default-dockercfg-xz768\""
Oct 29 22:55:25.184373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.184303    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-data\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.184557 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.184438    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-workload-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.184557 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.184480    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istiod-ca-cert\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.184674 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.184579    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gmtk\" (UniqueName: \"kubernetes.io/projected/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-kube-api-access-7gmtk\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.184674 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.184617    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-envoy\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.184674 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.184649    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-workload-certs\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.184850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.184685    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-token\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.184850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.184780    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-credential-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.184967 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.184862    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-podinfo\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.286505 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.286467    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-data\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.286736 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.286539    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-workload-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.286736 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.286559    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istiod-ca-cert\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.286736 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.286654    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gmtk\" (UniqueName: \"kubernetes.io/projected/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-kube-api-access-7gmtk\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.286736 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.286692    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-envoy\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.286736 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.286723    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-workload-certs\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.287124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.286764    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-token\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.287124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.286944    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-credential-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.287124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.286967    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-data\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.287124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.287042    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-podinfo\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.287392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.287302    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-credential-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.287793 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.287769    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-workload-certs\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.288521 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.288495    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istiod-ca-cert\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.289388 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.289359    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-workload-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.290182 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.290155    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-envoy\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.291325 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.291264    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-podinfo\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.308096 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.308066    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gmtk\" (UniqueName: \"kubernetes.io/projected/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-kube-api-access-7gmtk\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.308470 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.308447    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-token\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.470545 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.470441    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:25.805348 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.805313    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"]
Oct 29 22:55:25.805476 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:55:25.805357    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b7beb6_4444_4752_9f17_c5b76b8eb2f9.slice/crio-cce62d1b396439c6b4830aa549e29054af27e26c304884a3b812853051ebb784 WatchSource:0}: Error finding container cce62d1b396439c6b4830aa549e29054af27e26c304884a3b812853051ebb784: Status 404 returned error can't find the container with id cce62d1b396439c6b4830aa549e29054af27e26c304884a3b812853051ebb784
Oct 29 22:55:25.808356 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.808325    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:55:25.808430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.808393    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:55:25.808430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:25.808423    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:55:26.232971 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.232934    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"]
Oct 29 22:55:26.238620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.238587    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.264933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.263635    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"]
Oct 29 22:55:26.308375 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.306534    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-credential-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.308375 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.306600    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-data\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.308375 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.306643    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54nb9\" (UniqueName: \"kubernetes.io/projected/faec6b9a-7468-48ae-8364-1f1913ff7265-kube-api-access-54nb9\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.308375 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.306699    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-workload-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.308375 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.306852    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-envoy\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.308375 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.306879    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-token\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.308375 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.307032    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-workload-certs\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.308375 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.307084    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faec6b9a-7468-48ae-8364-1f1913ff7265-istiod-ca-cert\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.308375 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.307214    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-podinfo\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.410924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.410878    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-data\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.411339 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.411311    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54nb9\" (UniqueName: \"kubernetes.io/projected/faec6b9a-7468-48ae-8364-1f1913ff7265-kube-api-access-54nb9\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.411455 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.411366    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-workload-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.411455 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.411372    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-data\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.411544 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.411470    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-envoy\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.411544 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.411486    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-token\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.411622 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.411591    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-workload-certs\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.411710 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.411624    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faec6b9a-7468-48ae-8364-1f1913ff7265-istiod-ca-cert\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.411782 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.411703    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-podinfo\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.415833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.411945    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-credential-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.415833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.412261    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-workload-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.415833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.412380    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-credential-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.415833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.412639    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-workload-certs\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.415833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.413246    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faec6b9a-7468-48ae-8364-1f1913ff7265-istiod-ca-cert\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.416227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.415937    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-envoy\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.418328 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.418302    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-podinfo\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.461836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.444580    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54nb9\" (UniqueName: \"kubernetes.io/projected/faec6b9a-7468-48ae-8364-1f1913ff7265-kube-api-access-54nb9\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.466871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.466366    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-token\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.516848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.516735    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2" event={"ID":"29b7beb6-4444-4752-9f17-c5b76b8eb2f9","Type":"ContainerStarted","Data":"72d8e52e3e5ec1583ab3e989aa03f825c40095b772eca7904d4c1618dbebe1e4"}
Oct 29 22:55:26.516848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.516790    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2" event={"ID":"29b7beb6-4444-4752-9f17-c5b76b8eb2f9","Type":"ContainerStarted","Data":"cce62d1b396439c6b4830aa549e29054af27e26c304884a3b812853051ebb784"}
Oct 29 22:55:26.563879 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.563761    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2" podStartSLOduration=1.5637380589999998 podStartE2EDuration="1.563738059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:26.561641926 +0000 UTC m=+2465.473085483" watchObservedRunningTime="2026-10-29 22:55:26.563738059 +0000 UTC m=+2465.475181614"
Oct 29 22:55:26.570647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.570610    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:26.916178 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:55:26.916137    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaec6b9a_7468_48ae_8364_1f1913ff7265.slice/crio-ff1e2cee73998aabe6c6fd8ccf3c27cac298874165ffca884c306e46ca06526c WatchSource:0}: Error finding container ff1e2cee73998aabe6c6fd8ccf3c27cac298874165ffca884c306e46ca06526c: Status 404 returned error can't find the container with id ff1e2cee73998aabe6c6fd8ccf3c27cac298874165ffca884c306e46ca06526c
Oct 29 22:55:26.918970 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.918942    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"]
Oct 29 22:55:26.919334 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.919306    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:55:26.919380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.919369    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:55:26.919413 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:26.919399    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:55:27.476834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:27.476527    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 29 22:55:27.476834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:27.476637    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2" podUID="29b7beb6-4444-4752-9f17-c5b76b8eb2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 29 22:55:27.477134 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:27.477049    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:27.529403 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:27.529352    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k" event={"ID":"faec6b9a-7468-48ae-8364-1f1913ff7265","Type":"ContainerStarted","Data":"a481b7088306fa00a66ac1e1c467a4a7cc8392db788b0e7a23a078ec73539471"}
Oct 29 22:55:27.529855 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:27.529443    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k" event={"ID":"faec6b9a-7468-48ae-8364-1f1913ff7265","Type":"ContainerStarted","Data":"ff1e2cee73998aabe6c6fd8ccf3c27cac298874165ffca884c306e46ca06526c"}
Oct 29 22:55:27.570878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:27.570791    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:27.578158 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:27.578098    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k" podStartSLOduration=1.578078154 podStartE2EDuration="1.578078154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:27.573592256 +0000 UTC m=+2466.485035812" watchObservedRunningTime="2026-10-29 22:55:27.578078154 +0000 UTC m=+2466.489521711"
Oct 29 22:55:28.483862 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:28.478279    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 29 22:55:28.483862 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:28.478346    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2" podUID="29b7beb6-4444-4752-9f17-c5b76b8eb2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 29 22:55:28.587987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:28.579933    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:55:28.587987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:28.580051    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k" podUID="faec6b9a-7468-48ae-8364-1f1913ff7265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 29 22:55:29.472938 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:29.472903    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 29 22:55:29.473282 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:29.473254    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2" podUID="29b7beb6-4444-4752-9f17-c5b76b8eb2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 29 22:55:29.577837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:29.574830    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:55:29.577837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:29.574894    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k" podUID="faec6b9a-7468-48ae-8364-1f1913ff7265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 29 22:55:29.610961 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:29.609856    4559 generic.go:356] "Generic (PLEG): container finished" podID="7f3dffab-440d-4fb7-81bb-9c63ed54d908" containerID="ffc4eb0a936fcbd2f7d4753d6a50390117a552d7cf382444b7ad6bab7125ced2" exitCode=0
Oct 29 22:55:29.610961 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:29.610845    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks" event={"ID":"7f3dffab-440d-4fb7-81bb-9c63ed54d908","Type":"ContainerDied","Data":"ffc4eb0a936fcbd2f7d4753d6a50390117a552d7cf382444b7ad6bab7125ced2"}
Oct 29 22:55:30.003497 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.003415    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:55:30.041838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.039598    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-envoy\") pod \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") "
Oct 29 22:55:30.041838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.039799    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3dffab-440d-4fb7-81bb-9c63ed54d908-kube-api-access-wpbx8\" (UniqueName: \"kubernetes.io/projected/7f3dffab-440d-4fb7-81bb-9c63ed54d908-kube-api-access-wpbx8\") pod \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") "
Oct 29 22:55:30.041838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.039933    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-credential-socket\") pod \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") "
Oct 29 22:55:30.041838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.040063    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-token\" (UniqueName: \"kubernetes.io/projected/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-token\") pod \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") "
Oct 29 22:55:30.041838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.040107    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-podinfo\") pod \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") "
Oct 29 22:55:30.041838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.040157    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-data\") pod \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") "
Oct 29 22:55:30.041838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.040206    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-workload-certs\") pod \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") "
Oct 29 22:55:30.041838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.040248    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-workload-socket\") pod \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") "
Oct 29 22:55:30.041838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.040293    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istiod-ca-cert\") pod \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\" (UID: \"7f3dffab-440d-4fb7-81bb-9c63ed54d908\") "
Oct 29 22:55:30.056837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.053490    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-data" pod "7f3dffab-440d-4fb7-81bb-9c63ed54d908" (UID: "7f3dffab-440d-4fb7-81bb-9c63ed54d908"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:55:30.056837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.053566    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-workload-socket" pod "7f3dffab-440d-4fb7-81bb-9c63ed54d908" (UID: "7f3dffab-440d-4fb7-81bb-9c63ed54d908"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:55:30.056837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.053613    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-workload-certs" pod "7f3dffab-440d-4fb7-81bb-9c63ed54d908" (UID: "7f3dffab-440d-4fb7-81bb-9c63ed54d908"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:55:30.056837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.053656    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-credential-socket" pod "7f3dffab-440d-4fb7-81bb-9c63ed54d908" (UID: "7f3dffab-440d-4fb7-81bb-9c63ed54d908"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:55:30.056837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.054079    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istiod-ca-cert" pod "7f3dffab-440d-4fb7-81bb-9c63ed54d908" (UID: "7f3dffab-440d-4fb7-81bb-9c63ed54d908"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:30.105887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.071750    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3dffab-440d-4fb7-81bb-9c63ed54d908-kube-api-access-wpbx8" pod "7f3dffab-440d-4fb7-81bb-9c63ed54d908" (UID: "7f3dffab-440d-4fb7-81bb-9c63ed54d908"). InnerVolumeSpecName "kube-api-access-wpbx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:30.105887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.074455    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-podinfo" pod "7f3dffab-440d-4fb7-81bb-9c63ed54d908" (UID: "7f3dffab-440d-4fb7-81bb-9c63ed54d908"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:55:30.105887 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.088289    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-envoy" pod "7f3dffab-440d-4fb7-81bb-9c63ed54d908" (UID: "7f3dffab-440d-4fb7-81bb-9c63ed54d908"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:55:30.116144 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.116082    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-token" pod "7f3dffab-440d-4fb7-81bb-9c63ed54d908" (UID: "7f3dffab-440d-4fb7-81bb-9c63ed54d908"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:30.166837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.166485    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpbx8\" (UniqueName: \"kubernetes.io/projected/7f3dffab-440d-4fb7-81bb-9c63ed54d908-kube-api-access-wpbx8\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:55:30.166837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.166528    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:55:30.166837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.166543    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:55:30.166837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.166557    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:55:30.166837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.166571    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:55:30.166837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.166584    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:55:30.166837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.166597    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:55:30.166837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.166609    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:55:30.166837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.166620    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3dffab-440d-4fb7-81bb-9c63ed54d908-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:55:30.473836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.473230    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 29 22:55:30.473836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.473294    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2" podUID="29b7beb6-4444-4752-9f17-c5b76b8eb2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 29 22:55:30.575847 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.573254    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:55:30.575847 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.573321    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k" podUID="faec6b9a-7468-48ae-8364-1f1913ff7265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 29 22:55:30.778952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.770797    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks" event={"ID":"7f3dffab-440d-4fb7-81bb-9c63ed54d908","Type":"ContainerDied","Data":"a9eacf5668cdc107a0df44071aae5d77d419955d4be506fed25ca1b306693e14"}
Oct 29 22:55:30.778952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.770966    4559 scope.go:122] "RemoveContainer" containerID="ffc4eb0a936fcbd2f7d4753d6a50390117a552d7cf382444b7ad6bab7125ced2"
Oct 29 22:55:30.778952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.771209    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"
Oct 29 22:55:30.905990 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.905905    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"]
Oct 29 22:55:30.915236 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:30.915210    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-vwlks"]
Oct 29 22:55:31.261839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:31.261423    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f3dffab-440d-4fb7-81bb-9c63ed54d908" path="/var/lib/kubelet/pods/7f3dffab-440d-4fb7-81bb-9c63ed54d908/volumes"
Oct 29 22:55:31.471835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:31.471281    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 29 22:55:31.471835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:31.471345    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2" podUID="29b7beb6-4444-4752-9f17-c5b76b8eb2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 29 22:55:31.578153 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:31.578057    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:55:31.578405 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:31.578376    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k" podUID="faec6b9a-7468-48ae-8364-1f1913ff7265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 29 22:55:32.471841 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:32.471750    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 29 22:55:32.472291 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:32.471843    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2" podUID="29b7beb6-4444-4752-9f17-c5b76b8eb2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 29 22:55:32.571368 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:32.571334    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:55:32.571632 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:32.571610    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k" podUID="faec6b9a-7468-48ae-8364-1f1913ff7265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 29 22:55:33.476557 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:33.476516    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:33.477139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:33.477022    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:33.477797 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:33.477776    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 22:55:33.571388 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:33.571098    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:55:33.571388 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:33.571170    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k" podUID="faec6b9a-7468-48ae-8364-1f1913ff7265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 29 22:55:34.577005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:34.576971    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:34.577449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:34.577398    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:34.578178 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:34.578158    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 22:55:43.695866 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:55:43.695827    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:55:49.511896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:49.511857    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"]
Oct 29 22:55:49.522836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:49.513694    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7f3dffab-440d-4fb7-81bb-9c63ed54d908" containerName="istio-proxy"
Oct 29 22:55:49.524137 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:49.524100    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f3dffab-440d-4fb7-81bb-9c63ed54d908" containerName="istio-proxy"
Oct 29 22:55:49.524722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:49.524702    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f3dffab-440d-4fb7-81bb-9c63ed54d908" containerName="istio-proxy"
Oct 29 22:55:49.755626 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:49.755587    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:49.804149 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:49.800933    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4796a9-openshift-default-dockercfg-kr29c\""
Oct 29 22:55:49.835960 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:49.835909    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8" podUID="882dc67e-fe87-4033-8be0-bfdc48845531" containerName="main" probeResult="failure" output="Get \"https://10.134.0.207:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:49.930930 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:49.930880    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"]
Oct 29 22:55:49.995612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:49.994195    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-workload-certs\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:49.995612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:49.994312    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-podinfo\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:49.995612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:49.994342    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istiod-ca-cert\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:49.995612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:49.994379    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-workload-socket\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:49.995612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:49.994417    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-data\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:49.995612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:49.994444    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-envoy\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:49.995612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:49.994584    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-token\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:49.995612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:49.994621    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcd29\" (UniqueName: \"kubernetes.io/projected/ea391d71-881c-41fb-bb0d-ee9d03a681d0-kube-api-access-bcd29\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:49.995612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:49.994680    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-credential-socket\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.096471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.096373    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-podinfo\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.096471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.096413    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istiod-ca-cert\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.096471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.096443    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-workload-socket\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.096471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.096471    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-data\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.096858 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.096490    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-envoy\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.097006 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.096980    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-workload-socket\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.097134 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.097043    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-token\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.097134 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.097104    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcd29\" (UniqueName: \"kubernetes.io/projected/ea391d71-881c-41fb-bb0d-ee9d03a681d0-kube-api-access-bcd29\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.097258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.097150    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-credential-socket\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.097258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.097045    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-data\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.097258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.097192    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istiod-ca-cert\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.097445 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.097432    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-workload-certs\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.097533 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.097509    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-credential-socket\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.097740 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.097716    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-workload-certs\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.099903 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.099871    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-envoy\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.100035 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.100015    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-podinfo\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.116709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.116681    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-token\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.116896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.116784    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcd29\" (UniqueName: \"kubernetes.io/projected/ea391d71-881c-41fb-bb0d-ee9d03a681d0-kube-api-access-bcd29\") pod \"e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") " pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.256152 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.256117    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:50.586418 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:55:50.586390    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea391d71_881c_41fb_bb0d_ee9d03a681d0.slice/crio-2a64da2ea5820668f2f1bc95e46c02ada517bbd6622ab94cd464ea3b5f4a01f7 WatchSource:0}: Error finding container 2a64da2ea5820668f2f1bc95e46c02ada517bbd6622ab94cd464ea3b5f4a01f7: Status 404 returned error can't find the container with id 2a64da2ea5820668f2f1bc95e46c02ada517bbd6622ab94cd464ea3b5f4a01f7
Oct 29 22:55:50.586768 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.586547    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"]
Oct 29 22:55:50.589698 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.589667    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:55:50.589766 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.589745    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:55:50.589842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:50.589775    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:55:51.132888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:51.132847    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv" event={"ID":"ea391d71-881c-41fb-bb0d-ee9d03a681d0","Type":"ContainerStarted","Data":"7b01695e3676eae49a793a3963e36449026fef50438f5c10a7271ed37e14c966"}
Oct 29 22:55:51.133084 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:51.132921    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv" event={"ID":"ea391d71-881c-41fb-bb0d-ee9d03a681d0","Type":"ContainerStarted","Data":"2a64da2ea5820668f2f1bc95e46c02ada517bbd6622ab94cd464ea3b5f4a01f7"}
Oct 29 22:55:51.192589 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:51.192523    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv" podStartSLOduration=2.192502812 podStartE2EDuration="2.192502812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:51.191782501 +0000 UTC m=+2490.103226056" watchObservedRunningTime="2026-10-29 22:55:51.192502812 +0000 UTC m=+2490.103946370"
Oct 29 22:55:51.256985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:51.256944    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:51.263654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:51.263616    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:52.142954 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:52.142924    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:52.144265 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:52.144235    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:55:53.003417 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:53.003382    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"]
Oct 29 22:55:53.039311 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:53.039240    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"]
Oct 29 22:55:55.569624 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:55.569583    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"]
Oct 29 22:55:55.576570 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:55.576538    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"
Oct 29 22:55:55.601101 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:55.601075    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4796a9\""
Oct 29 22:55:55.603064 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:55.603032    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"]
Oct 29 22:55:55.628566 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:55.628529    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4796a9-dockercfg-jjrhz\""
Oct 29 22:55:55.759871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:55.736033    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6594f54e-cae8-43df-a4c7-8096eaee4a70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b\" (UID: \"6594f54e-cae8-43df-a4c7-8096eaee4a70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"
Oct 29 22:55:55.761608 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:55.761569    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhps6\" (UniqueName: \"kubernetes.io/projected/6594f54e-cae8-43df-a4c7-8096eaee4a70-kube-api-access-lhps6\") pod \"payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b\" (UID: \"6594f54e-cae8-43df-a4c7-8096eaee4a70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"
Oct 29 22:55:55.893612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:55.891052    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhps6\" (UniqueName: \"kubernetes.io/projected/6594f54e-cae8-43df-a4c7-8096eaee4a70-kube-api-access-lhps6\") pod \"payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b\" (UID: \"6594f54e-cae8-43df-a4c7-8096eaee4a70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"
Oct 29 22:55:55.893612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:55.891355    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6594f54e-cae8-43df-a4c7-8096eaee4a70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b\" (UID: \"6594f54e-cae8-43df-a4c7-8096eaee4a70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"
Oct 29 22:55:55.893612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:55.892436    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6594f54e-cae8-43df-a4c7-8096eaee4a70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b\" (UID: \"6594f54e-cae8-43df-a4c7-8096eaee4a70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"
Oct 29 22:55:55.984987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:55.984896    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhps6\" (UniqueName: \"kubernetes.io/projected/6594f54e-cae8-43df-a4c7-8096eaee4a70-kube-api-access-lhps6\") pod \"payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b\" (UID: \"6594f54e-cae8-43df-a4c7-8096eaee4a70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"
Oct 29 22:55:56.210099 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:56.209999    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"
Oct 29 22:55:56.812848 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:55:56.812742    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6594f54e_cae8_43df_a4c7_8096eaee4a70.slice/crio-e4547023db436f5a29cd6c0a4209455f4216ed1ea869acc2aad0a6e88074e5c7 WatchSource:0}: Error finding container e4547023db436f5a29cd6c0a4209455f4216ed1ea869acc2aad0a6e88074e5c7: Status 404 returned error can't find the container with id e4547023db436f5a29cd6c0a4209455f4216ed1ea869acc2aad0a6e88074e5c7
Oct 29 22:55:56.813373 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:56.812973    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"]
Oct 29 22:55:57.214502 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:57.214466    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b" event={"ID":"6594f54e-cae8-43df-a4c7-8096eaee4a70","Type":"ContainerStarted","Data":"ead5f990bb684821ef3a657936c4db8bbcd1f0be95482394fbd8a55bb4362f20"}
Oct 29 22:55:57.214679 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:57.214523    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b" event={"ID":"6594f54e-cae8-43df-a4c7-8096eaee4a70","Type":"ContainerStarted","Data":"e4547023db436f5a29cd6c0a4209455f4216ed1ea869acc2aad0a6e88074e5c7"}
Oct 29 22:55:57.214679 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:57.214578    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"
Oct 29 22:55:57.262706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:57.262642    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b" podStartSLOduration=2.262623225 podStartE2EDuration="2.262623225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:57.253950546 +0000 UTC m=+2496.165394101" watchObservedRunningTime="2026-10-29 22:55:57.262623225 +0000 UTC m=+2496.174066779"
Oct 29 22:55:58.004276 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.004233    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj" podUID="bae036a5-0fa1-4aca-8cb8-6b2982c994cf" containerName="payload-processing" containerID="cri-o://70cbb3421f08fffb58dcf6f81e2eaef4a8df3d0ad2940817a1e70df27de6493e" gracePeriod=25
Oct 29 22:55:58.040139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.040095    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928" podUID="4ad36d06-a1b2-4498-8a82-539ff6addb6a" containerName="payload-pre-processing" containerID="cri-o://c372f83448fb50ee70a73b6cc0d5eaf4da1e63f99c21bcc24dda9a1364b87d31" gracePeriod=25
Oct 29 22:55:58.228935 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.228633    4559 generic.go:356] "Generic (PLEG): container finished" podID="bae036a5-0fa1-4aca-8cb8-6b2982c994cf" containerID="70cbb3421f08fffb58dcf6f81e2eaef4a8df3d0ad2940817a1e70df27de6493e" exitCode=0
Oct 29 22:55:58.228935 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.228797    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj" event={"ID":"bae036a5-0fa1-4aca-8cb8-6b2982c994cf","Type":"ContainerDied","Data":"70cbb3421f08fffb58dcf6f81e2eaef4a8df3d0ad2940817a1e70df27de6493e"}
Oct 29 22:55:58.231181 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.231145    4559 generic.go:356] "Generic (PLEG): container finished" podID="4ad36d06-a1b2-4498-8a82-539ff6addb6a" containerID="c372f83448fb50ee70a73b6cc0d5eaf4da1e63f99c21bcc24dda9a1364b87d31" exitCode=0
Oct 29 22:55:58.231400 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.231373    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928" event={"ID":"4ad36d06-a1b2-4498-8a82-539ff6addb6a","Type":"ContainerDied","Data":"c372f83448fb50ee70a73b6cc0d5eaf4da1e63f99c21bcc24dda9a1364b87d31"}
Oct 29 22:55:58.382425 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.382393    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"
Oct 29 22:55:58.412349 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.412323    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"
Oct 29 22:55:58.423888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.423845    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae036a5-0fa1-4aca-8cb8-6b2982c994cf-kube-api-access-f8p4z\" (UniqueName: \"kubernetes.io/projected/bae036a5-0fa1-4aca-8cb8-6b2982c994cf-kube-api-access-f8p4z\") pod \"bae036a5-0fa1-4aca-8cb8-6b2982c994cf\" (UID: \"bae036a5-0fa1-4aca-8cb8-6b2982c994cf\") "
Oct 29 22:55:58.424571 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.424540    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae036a5-0fa1-4aca-8cb8-6b2982c994cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae036a5-0fa1-4aca-8cb8-6b2982c994cf-plugins-config-volume\") pod \"bae036a5-0fa1-4aca-8cb8-6b2982c994cf\" (UID: \"bae036a5-0fa1-4aca-8cb8-6b2982c994cf\") "
Oct 29 22:55:58.426402 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.425007    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae036a5-0fa1-4aca-8cb8-6b2982c994cf-plugins-config-volume" pod "bae036a5-0fa1-4aca-8cb8-6b2982c994cf" (UID: "bae036a5-0fa1-4aca-8cb8-6b2982c994cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:58.426402 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.425436    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae036a5-0fa1-4aca-8cb8-6b2982c994cf-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:55:58.426632 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.426466    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae036a5-0fa1-4aca-8cb8-6b2982c994cf-kube-api-access-f8p4z" pod "bae036a5-0fa1-4aca-8cb8-6b2982c994cf" (UID: "bae036a5-0fa1-4aca-8cb8-6b2982c994cf"). InnerVolumeSpecName "kube-api-access-f8p4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:58.526501 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.526468    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad36d06-a1b2-4498-8a82-539ff6addb6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad36d06-a1b2-4498-8a82-539ff6addb6a-plugins-config-volume\") pod \"4ad36d06-a1b2-4498-8a82-539ff6addb6a\" (UID: \"4ad36d06-a1b2-4498-8a82-539ff6addb6a\") "
Oct 29 22:55:58.526925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.526901    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad36d06-a1b2-4498-8a82-539ff6addb6a-kube-api-access-c569s\" (UniqueName: \"kubernetes.io/projected/4ad36d06-a1b2-4498-8a82-539ff6addb6a-kube-api-access-c569s\") pod \"4ad36d06-a1b2-4498-8a82-539ff6addb6a\" (UID: \"4ad36d06-a1b2-4498-8a82-539ff6addb6a\") "
Oct 29 22:55:58.526982 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.526942    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad36d06-a1b2-4498-8a82-539ff6addb6a-plugins-config-volume" pod "4ad36d06-a1b2-4498-8a82-539ff6addb6a" (UID: "4ad36d06-a1b2-4498-8a82-539ff6addb6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:58.527550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.527527    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad36d06-a1b2-4498-8a82-539ff6addb6a-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:55:58.527550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.527547    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8p4z\" (UniqueName: \"kubernetes.io/projected/bae036a5-0fa1-4aca-8cb8-6b2982c994cf-kube-api-access-f8p4z\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:55:58.529132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.529103    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad36d06-a1b2-4498-8a82-539ff6addb6a-kube-api-access-c569s" pod "4ad36d06-a1b2-4498-8a82-539ff6addb6a" (UID: "4ad36d06-a1b2-4498-8a82-539ff6addb6a"). InnerVolumeSpecName "kube-api-access-c569s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:58.629091 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:58.629005    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c569s\" (UniqueName: \"kubernetes.io/projected/4ad36d06-a1b2-4498-8a82-539ff6addb6a-kube-api-access-c569s\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:55:59.244021 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:59.243993    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"
Oct 29 22:55:59.244377 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:59.244022    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj" event={"ID":"bae036a5-0fa1-4aca-8cb8-6b2982c994cf","Type":"ContainerDied","Data":"7389bc03eff4284e9caf4bb9efc0617024b1c57d804790c732a6687fd4f217b2"}
Oct 29 22:55:59.244377 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:59.244093    4559 scope.go:122] "RemoveContainer" containerID="70cbb3421f08fffb58dcf6f81e2eaef4a8df3d0ad2940817a1e70df27de6493e"
Oct 29 22:55:59.245799 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:59.245778    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"
Oct 29 22:55:59.245944 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:59.245843    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928" event={"ID":"4ad36d06-a1b2-4498-8a82-539ff6addb6a","Type":"ContainerDied","Data":"d83d693a4101098d9bc5064299b9d73c0e717a023c7f31b603c169f9739335ec"}
Oct 29 22:55:59.257758 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:59.257734    4559 scope.go:122] "RemoveContainer" containerID="c372f83448fb50ee70a73b6cc0d5eaf4da1e63f99c21bcc24dda9a1364b87d31"
Oct 29 22:55:59.302593 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:59.302492    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"]
Oct 29 22:55:59.306877 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:59.306845    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a962d7-7cf57bb744-49pdj"]
Oct 29 22:55:59.348366 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:59.348327    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"]
Oct 29 22:55:59.354542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:55:59.354510    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-64fcd788bc-vc928"]
Oct 29 22:56:01.108876 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.102016    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"]
Oct 29 22:56:01.108876 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.103434    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bae036a5-0fa1-4aca-8cb8-6b2982c994cf" containerName="payload-processing"
Oct 29 22:56:01.108876 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.103492    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae036a5-0fa1-4aca-8cb8-6b2982c994cf" containerName="payload-processing"
Oct 29 22:56:01.108876 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.103550    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4ad36d06-a1b2-4498-8a82-539ff6addb6a" containerName="payload-pre-processing"
Oct 29 22:56:01.108876 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.103559    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad36d06-a1b2-4498-8a82-539ff6addb6a" containerName="payload-pre-processing"
Oct 29 22:56:01.108876 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.103748    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae036a5-0fa1-4aca-8cb8-6b2982c994cf" containerName="payload-processing"
Oct 29 22:56:01.108876 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.103764    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad36d06-a1b2-4498-8a82-539ff6addb6a" containerName="payload-pre-processing"
Oct 29 22:56:01.113140 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.113113    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"
Oct 29 22:56:01.132245 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.132193    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"]
Oct 29 22:56:01.141303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.141137    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7be71d\""
Oct 29 22:56:01.141303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.141268    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7be71d-dockercfg-lfrlr\""
Oct 29 22:56:01.497440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.497400    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad36d06-a1b2-4498-8a82-539ff6addb6a" path="/var/lib/kubelet/pods/4ad36d06-a1b2-4498-8a82-539ff6addb6a/volumes"
Oct 29 22:56:01.498204 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.498179    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae036a5-0fa1-4aca-8cb8-6b2982c994cf" path="/var/lib/kubelet/pods/bae036a5-0fa1-4aca-8cb8-6b2982c994cf/volumes"
Oct 29 22:56:01.542371 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.542327    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h5b2\" (UniqueName: \"kubernetes.io/projected/00aa21ab-c4a0-460c-9647-ba3fece700cc-kube-api-access-6h5b2\") pod \"payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d\" (UID: \"00aa21ab-c4a0-460c-9647-ba3fece700cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"
Oct 29 22:56:01.542591 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.542388    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00aa21ab-c4a0-460c-9647-ba3fece700cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d\" (UID: \"00aa21ab-c4a0-460c-9647-ba3fece700cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"
Oct 29 22:56:01.644984 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.644931    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h5b2\" (UniqueName: \"kubernetes.io/projected/00aa21ab-c4a0-460c-9647-ba3fece700cc-kube-api-access-6h5b2\") pod \"payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d\" (UID: \"00aa21ab-c4a0-460c-9647-ba3fece700cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"
Oct 29 22:56:01.645237 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.645000    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00aa21ab-c4a0-460c-9647-ba3fece700cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d\" (UID: \"00aa21ab-c4a0-460c-9647-ba3fece700cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"
Oct 29 22:56:01.645898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.645875    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00aa21ab-c4a0-460c-9647-ba3fece700cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d\" (UID: \"00aa21ab-c4a0-460c-9647-ba3fece700cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"
Oct 29 22:56:01.664547 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.664512    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h5b2\" (UniqueName: \"kubernetes.io/projected/00aa21ab-c4a0-460c-9647-ba3fece700cc-kube-api-access-6h5b2\") pod \"payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d\" (UID: \"00aa21ab-c4a0-460c-9647-ba3fece700cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"
Oct 29 22:56:01.771947 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:01.767276    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"
Oct 29 22:56:02.129030 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:02.129000    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"]
Oct 29 22:56:02.129454 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:56:02.129427    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00aa21ab_c4a0_460c_9647_ba3fece700cc.slice/crio-dab1505cca6d9cc8c6f462071b323dd13ce9623787af0e033ad696a93cd0263d WatchSource:0}: Error finding container dab1505cca6d9cc8c6f462071b323dd13ce9623787af0e033ad696a93cd0263d: Status 404 returned error can't find the container with id dab1505cca6d9cc8c6f462071b323dd13ce9623787af0e033ad696a93cd0263d
Oct 29 22:56:02.503573 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:02.503534    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d" event={"ID":"00aa21ab-c4a0-460c-9647-ba3fece700cc","Type":"ContainerStarted","Data":"dab1505cca6d9cc8c6f462071b323dd13ce9623787af0e033ad696a93cd0263d"}
Oct 29 22:56:04.568763 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:04.568719    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d" event={"ID":"00aa21ab-c4a0-460c-9647-ba3fece700cc","Type":"ContainerStarted","Data":"3933d302c9f33235ca7fa5b68cab508498d15014e84cb6952a796c14ffcbcbbf"}
Oct 29 22:56:04.569466 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:04.569445    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"
Oct 29 22:56:04.630372 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:04.630312    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d" podStartSLOduration=1.9807447470000001 podStartE2EDuration="3.630297851s" totalImagesPullingTime="1.649553104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:04.627110602 +0000 UTC m=+2503.538554147" watchObservedRunningTime="2026-10-29 22:56:04.630297851 +0000 UTC m=+2503.541741404"
Oct 29 22:56:08.235954 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:08.235920    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"
Oct 29 22:56:10.027712 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.027673    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"]
Oct 29 22:56:10.034020 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.034000    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.045314 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.045284    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"]
Oct 29 22:56:10.052871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.052796    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.055759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.055729    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"]
Oct 29 22:56:10.087023 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.086987    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"]
Oct 29 22:56:10.181753 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.181712    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.182014 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.181782    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.182014 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.181933    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.182014 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.181983    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.182154 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.182060    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.182154 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.182094    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.182233 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.182150    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.182233 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.182189    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zswj4\" (UniqueName: \"kubernetes.io/projected/fed652a5-91ae-4f75-975a-2d858e4c3bb9-kube-api-access-zswj4\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.182328 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.182281    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.182386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.182331    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.182461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.182430    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjlpx\" (UniqueName: \"kubernetes.io/projected/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-kube-api-access-hjlpx\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.182524 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.182475    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.182524 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.182511    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.182646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.182583    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.182646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.182625    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.182739 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.182668    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.182739 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.182691    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.182843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.182833    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.284739 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.284650    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.284739 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.284710    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.285095 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.284756    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.285357 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.285314    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.285475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.285313    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.285475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.284801    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.285475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.285434    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.285475 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.285469    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.285776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.285500    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.285776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.285553    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zswj4\" (UniqueName: \"kubernetes.io/projected/fed652a5-91ae-4f75-975a-2d858e4c3bb9-kube-api-access-zswj4\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.285776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.285619    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.286077 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.285912    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.286333 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.286310    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.286447 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.286376    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.286447 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.286421    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.286592 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.286479    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjlpx\" (UniqueName: \"kubernetes.io/projected/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-kube-api-access-hjlpx\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.286592 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.286509    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.286592 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.286528    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.286592 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.286541    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.286592 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.286562    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.286592 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.286585    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.287042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.286633    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.287042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.286657    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.287042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.286744    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.287202 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.287057    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.287202 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.287167    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.287328 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.287209    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.287402 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.287358    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.288412 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.288376    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.289407 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.289386    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.289407 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.289397    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.289550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.289496    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.310402 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.310370    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjlpx\" (UniqueName: \"kubernetes.io/projected/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-kube-api-access-hjlpx\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.310802 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.310773    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zswj4\" (UniqueName: \"kubernetes.io/projected/fed652a5-91ae-4f75-975a-2d858e4c3bb9-kube-api-access-zswj4\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.311347 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.311326    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.311512 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.311493    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.362091 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.362055    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:10.409008 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.408967    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:10.689386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.689354    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"]
Oct 29 22:56:10.690116 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:56:10.690085    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064c3a9c_29ac_48fa_9382_1a8ac20fb18a.slice/crio-e6a1d4f565abe2fd0d76c2781b4cec9fecebe6ac3c7b36db2d64945d382dc1c3 WatchSource:0}: Error finding container e6a1d4f565abe2fd0d76c2781b4cec9fecebe6ac3c7b36db2d64945d382dc1c3: Status 404 returned error can't find the container with id e6a1d4f565abe2fd0d76c2781b4cec9fecebe6ac3c7b36db2d64945d382dc1c3
Oct 29 22:56:10.693441 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.693404    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:56:10.693539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.693479    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:56:10.693539 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.693509    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:56:10.709709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.709675    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv" event={"ID":"064c3a9c-29ac-48fa-9382-1a8ac20fb18a","Type":"ContainerStarted","Data":"e6a1d4f565abe2fd0d76c2781b4cec9fecebe6ac3c7b36db2d64945d382dc1c3"}
Oct 29 22:56:10.757864 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:56:10.757440    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed652a5_91ae_4f75_975a_2d858e4c3bb9.slice/crio-8a1d52d654e7fc0488dba916e892bca19c243652f2a489dc958f6833a479827d WatchSource:0}: Error finding container 8a1d52d654e7fc0488dba916e892bca19c243652f2a489dc958f6833a479827d: Status 404 returned error can't find the container with id 8a1d52d654e7fc0488dba916e892bca19c243652f2a489dc958f6833a479827d
Oct 29 22:56:10.758602 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.758574    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"]
Oct 29 22:56:10.761596 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.761559    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:56:10.761721 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.761647    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:56:10.761721 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:10.761691    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:56:11.723888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:11.723491    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc" event={"ID":"fed652a5-91ae-4f75-975a-2d858e4c3bb9","Type":"ContainerStarted","Data":"bad7f05fe49b646480a86cc09be7f0fb9cbd43463dbfe3e201ad207e83bb96ea"}
Oct 29 22:56:11.723888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:11.723558    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc" event={"ID":"fed652a5-91ae-4f75-975a-2d858e4c3bb9","Type":"ContainerStarted","Data":"8a1d52d654e7fc0488dba916e892bca19c243652f2a489dc958f6833a479827d"}
Oct 29 22:56:11.726212 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:11.726182    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv" event={"ID":"064c3a9c-29ac-48fa-9382-1a8ac20fb18a","Type":"ContainerStarted","Data":"790a0605487c3f254abcc23b640adc592e74ee95ac7f6827d7f5aa466b12e912"}
Oct 29 22:56:11.807831 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:11.807639    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc" podStartSLOduration=1.8076159349999998 podStartE2EDuration="1.807615935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:11.787350672 +0000 UTC m=+2510.698794227" watchObservedRunningTime="2026-10-29 22:56:11.807615935 +0000 UTC m=+2510.719059482"
Oct 29 22:56:11.859631 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:11.859552    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv" podStartSLOduration=2.859525845 podStartE2EDuration="2.859525845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:11.832467573 +0000 UTC m=+2510.743911124" watchObservedRunningTime="2026-10-29 22:56:11.859525845 +0000 UTC m=+2510.770969398"
Oct 29 22:56:12.363836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:12.363297    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:12.380875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:12.366975    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 29 22:56:12.380875 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:12.367038    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv" podUID="064c3a9c-29ac-48fa-9382-1a8ac20fb18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 29 22:56:12.409604 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:12.409296    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:12.411161 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:12.411130    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 29 22:56:12.411301 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:12.411176    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc" podUID="fed652a5-91ae-4f75-975a-2d858e4c3bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 29 22:56:13.368839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:13.367970    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 29 22:56:13.368839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:13.368036    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv" podUID="064c3a9c-29ac-48fa-9382-1a8ac20fb18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 29 22:56:13.410242 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:13.410199    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 29 22:56:13.410422 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:13.410271    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc" podUID="fed652a5-91ae-4f75-975a-2d858e4c3bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 29 22:56:13.787838 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:56:13.776020    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:56:14.365320 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:14.365282    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 29 22:56:14.365610 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:14.365581    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv" podUID="064c3a9c-29ac-48fa-9382-1a8ac20fb18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 29 22:56:14.410291 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:14.410249    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 29 22:56:14.410834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:14.410785    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc" podUID="fed652a5-91ae-4f75-975a-2d858e4c3bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 29 22:56:15.412132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:15.412093    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 29 22:56:15.436531 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:15.436489    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv" podUID="064c3a9c-29ac-48fa-9382-1a8ac20fb18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 29 22:56:15.436873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:15.413254    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 29 22:56:15.437029 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:15.436911    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc" podUID="fed652a5-91ae-4f75-975a-2d858e4c3bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 29 22:56:16.365751 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:16.365712    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 29 22:56:16.366051 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:16.366019    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv" podUID="064c3a9c-29ac-48fa-9382-1a8ac20fb18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 29 22:56:16.409976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:16.409937    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 29 22:56:16.410173 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:16.409996    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc" podUID="fed652a5-91ae-4f75-975a-2d858e4c3bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 29 22:56:16.673048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:16.673009    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"
Oct 29 22:56:17.387837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:17.382837    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 29 22:56:17.387837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:17.383691    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv" podUID="064c3a9c-29ac-48fa-9382-1a8ac20fb18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 29 22:56:17.410257 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:17.410220    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 29 22:56:17.410529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:17.410502    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc" podUID="fed652a5-91ae-4f75-975a-2d858e4c3bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 29 22:56:18.369056 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:18.369023    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:18.369593 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:18.369567    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:18.370515 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:18.370490    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 22:56:18.415347 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:18.415317    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:18.415659 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:18.415631    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:18.416405 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:18.416387    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 22:56:19.837144 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:19.837114    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"]
Oct 29 22:56:20.048333 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.048290    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"]
Oct 29 22:56:20.055310 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.055278    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.072933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.072878    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-85862e-openshift-default-dockercfg-m7q24\""
Oct 29 22:56:20.090836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.090735    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"]
Oct 29 22:56:20.188644 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.188583    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-podinfo\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.188922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.188651    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-envoy\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.188922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.188758    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istiod-ca-cert\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.188922 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.188796    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-credential-socket\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.189143 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.189017    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5btx\" (UniqueName: \"kubernetes.io/projected/74383eae-9fc4-44f4-9279-a2c085c4eeb9-kube-api-access-b5btx\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.189143 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.189094    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-workload-certs\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.189264 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.189211    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-token\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.189320 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.189259    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-data\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.189380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.189354    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-workload-socket\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.290877 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.290840    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-podinfo\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.291096 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.290887    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-envoy\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.291096 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.290914    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istiod-ca-cert\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.291096 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.290970    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-credential-socket\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.291292 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.291160    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5btx\" (UniqueName: \"kubernetes.io/projected/74383eae-9fc4-44f4-9279-a2c085c4eeb9-kube-api-access-b5btx\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.291292 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.291242    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-workload-certs\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.291465 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.291435    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-credential-socket\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.291547 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.291448    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-token\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.291612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.291551    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-data\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.291706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.291671    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-workload-socket\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.292042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.292020    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-data\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.292155 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.292129    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-workload-certs\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.292422 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.292397    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-workload-socket\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.292801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.292777    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istiod-ca-cert\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.293893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.293867    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-podinfo\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.294722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.294704    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-envoy\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.329360 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.322368    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5btx\" (UniqueName: \"kubernetes.io/projected/74383eae-9fc4-44f4-9279-a2c085c4eeb9-kube-api-access-b5btx\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.348237 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.348115    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-token\") pod \"e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") " pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:20.386835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:20.384248    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:21.011620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:21.011523    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"]
Oct 29 22:56:21.020200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:21.020162    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:56:21.020318 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:21.020238    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:56:21.020318 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:21.020277    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:56:21.934491 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:21.934454    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm" event={"ID":"74383eae-9fc4-44f4-9279-a2c085c4eeb9","Type":"ContainerStarted","Data":"c3a133d3083c54ae82895401079bdcdd5a7276abbfec38ddcccc7ff872dd37d1"}
Oct 29 22:56:21.934704 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:21.934514    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm" event={"ID":"74383eae-9fc4-44f4-9279-a2c085c4eeb9","Type":"ContainerStarted","Data":"edffb347c075fab401bdd3286a3aa6b74e702bbbf7dfc5dbf8f538fbcd0db55a"}
Oct 29 22:56:21.994851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:21.994751    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm" podStartSLOduration=1.9947349079999999 podStartE2EDuration="1.994734908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:21.977946098 +0000 UTC m=+2520.889389690" watchObservedRunningTime="2026-10-29 22:56:21.994734908 +0000 UTC m=+2520.906178473"
Oct 29 22:56:22.385236 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:22.385151    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:22.392097 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:22.392068    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:22.942597 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:22.942565    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:22.943615 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:22.943591    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:56:32.326925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.326872    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"]
Oct 29 22:56:32.358846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.355285    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"
Oct 29 22:56:32.394437 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.394399    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8c4b02-dockercfg-cj8z5\""
Oct 29 22:56:32.394846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.394798    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8c4b02\""
Oct 29 22:56:32.552911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.518965    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f\" (UID: \"08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"
Oct 29 22:56:32.552911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.519172    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqjwk\" (UniqueName: \"kubernetes.io/projected/08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3-kube-api-access-cqjwk\") pod \"payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f\" (UID: \"08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"
Oct 29 22:56:32.573493 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.573339    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"]
Oct 29 22:56:32.603342 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.603250    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"]
Oct 29 22:56:32.610136 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.610109    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"
Oct 29 22:56:32.614674 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.614638    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"]
Oct 29 22:56:32.620750 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.620653    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f\" (UID: \"08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"
Oct 29 22:56:32.620972 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.620844    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqjwk\" (UniqueName: \"kubernetes.io/projected/08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3-kube-api-access-cqjwk\") pod \"payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f\" (UID: \"08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"
Oct 29 22:56:32.622506 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.621861    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f\" (UID: \"08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"
Oct 29 22:56:32.640673 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.640642    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqjwk\" (UniqueName: \"kubernetes.io/projected/08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3-kube-api-access-cqjwk\") pod \"payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f\" (UID: \"08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"
Oct 29 22:56:32.722333 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.722286    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4875d982-49ff-4402-86c3-e7a9f97ffba7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf\" (UID: \"4875d982-49ff-4402-86c3-e7a9f97ffba7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"
Oct 29 22:56:32.722863 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.722780    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc2mp\" (UniqueName: \"kubernetes.io/projected/4875d982-49ff-4402-86c3-e7a9f97ffba7-kube-api-access-vc2mp\") pod \"payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf\" (UID: \"4875d982-49ff-4402-86c3-e7a9f97ffba7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"
Oct 29 22:56:32.824679 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.824637    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4875d982-49ff-4402-86c3-e7a9f97ffba7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf\" (UID: \"4875d982-49ff-4402-86c3-e7a9f97ffba7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"
Oct 29 22:56:32.824893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.824868    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc2mp\" (UniqueName: \"kubernetes.io/projected/4875d982-49ff-4402-86c3-e7a9f97ffba7-kube-api-access-vc2mp\") pod \"payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf\" (UID: \"4875d982-49ff-4402-86c3-e7a9f97ffba7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"
Oct 29 22:56:32.825703 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.825679    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4875d982-49ff-4402-86c3-e7a9f97ffba7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf\" (UID: \"4875d982-49ff-4402-86c3-e7a9f97ffba7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"
Oct 29 22:56:32.844853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.844799    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc2mp\" (UniqueName: \"kubernetes.io/projected/4875d982-49ff-4402-86c3-e7a9f97ffba7-kube-api-access-vc2mp\") pod \"payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf\" (UID: \"4875d982-49ff-4402-86c3-e7a9f97ffba7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"
Oct 29 22:56:32.886571 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.886479    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"
Oct 29 22:56:32.930052 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:32.930008    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"
Oct 29 22:56:33.250110 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:33.250080    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"]
Oct 29 22:56:33.292623 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:56:33.292595    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4875d982_49ff_4402_86c3_e7a9f97ffba7.slice/crio-90506a35e50840b2b3950bb00bed4cc82a030d3500e776f69713bcac355c304a WatchSource:0}: Error finding container 90506a35e50840b2b3950bb00bed4cc82a030d3500e776f69713bcac355c304a: Status 404 returned error can't find the container with id 90506a35e50840b2b3950bb00bed4cc82a030d3500e776f69713bcac355c304a
Oct 29 22:56:33.292918 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:33.292888    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"]
Oct 29 22:56:34.234784 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:34.234414    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f" event={"ID":"08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3","Type":"ContainerStarted","Data":"ab00e83b955d42298684de19e585846733b898aefb1c53b77950f87f910658a7"}
Oct 29 22:56:34.234784 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:34.234477    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f" event={"ID":"08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3","Type":"ContainerStarted","Data":"b2d1cf73126dcc5d96c14d7a4687d0f08d095bf7e7dcccf369bf4f7380b520dc"}
Oct 29 22:56:34.239034 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:34.235658    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"
Oct 29 22:56:34.251910 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:34.251879    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf" event={"ID":"4875d982-49ff-4402-86c3-e7a9f97ffba7","Type":"ContainerStarted","Data":"bc0a99739e8bc6cae10def6163e916be9ed593721a2d4df66ad707a65eb7195f"}
Oct 29 22:56:34.252157 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:34.252140    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf" event={"ID":"4875d982-49ff-4402-86c3-e7a9f97ffba7","Type":"ContainerStarted","Data":"90506a35e50840b2b3950bb00bed4cc82a030d3500e776f69713bcac355c304a"}
Oct 29 22:56:34.253161 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:34.253131    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"
Oct 29 22:56:34.281636 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:34.281574    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f" podStartSLOduration=2.2815583139999998 podStartE2EDuration="2.281558314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:34.277524285 +0000 UTC m=+2533.188967839" watchObservedRunningTime="2026-10-29 22:56:34.281558314 +0000 UTC m=+2533.193001867"
Oct 29 22:56:34.322794 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:34.322742    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf" podStartSLOduration=2.322726875 podStartE2EDuration="2.322726875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:34.316744042 +0000 UTC m=+2533.228187595" watchObservedRunningTime="2026-10-29 22:56:34.322726875 +0000 UTC m=+2533.234170428"
Oct 29 22:56:35.032746 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.032686    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8" podUID="882dc67e-fe87-4033-8be0-bfdc48845531" containerName="main" containerID="cri-o://201536958e6ab2e917cc2390560277bee542945baa4d703a34fc3f7e9d3b9a79" gracePeriod=45
Oct 29 22:56:35.079918 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.079500    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8" podUID="882dc67e-fe87-4033-8be0-bfdc48845531" containerName="main" probeResult="failure" output="Get \"https://10.134.0.207:8000/ready\": dial tcp 10.134.0.207:8000: connect: connection refused"
Oct 29 22:56:35.298055 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.297966    4559 generic.go:356] "Generic (PLEG): container finished" podID="882dc67e-fe87-4033-8be0-bfdc48845531" containerID="201536958e6ab2e917cc2390560277bee542945baa4d703a34fc3f7e9d3b9a79" exitCode=0
Oct 29 22:56:35.299924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.299871    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8" event={"ID":"882dc67e-fe87-4033-8be0-bfdc48845531","Type":"ContainerDied","Data":"201536958e6ab2e917cc2390560277bee542945baa4d703a34fc3f7e9d3b9a79"}
Oct 29 22:56:35.580051 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.563414    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:56:35.670658 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.670605    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-home\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-home\") pod \"882dc67e-fe87-4033-8be0-bfdc48845531\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") "
Oct 29 22:56:35.696885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.696128    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882dc67e-fe87-4033-8be0-bfdc48845531-kube-api-access-jcxnk\" (UniqueName: \"kubernetes.io/projected/882dc67e-fe87-4033-8be0-bfdc48845531-kube-api-access-jcxnk\") pod \"882dc67e-fe87-4033-8be0-bfdc48845531\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") "
Oct 29 22:56:35.696885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.681890    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-home" pod "882dc67e-fe87-4033-8be0-bfdc48845531" (UID: "882dc67e-fe87-4033-8be0-bfdc48845531"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:56:35.696885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.696242    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-tmp-dir\") pod \"882dc67e-fe87-4033-8be0-bfdc48845531\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") "
Oct 29 22:56:35.696885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.696433    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/882dc67e-fe87-4033-8be0-bfdc48845531-tls-certs\" (UniqueName: \"kubernetes.io/secret/882dc67e-fe87-4033-8be0-bfdc48845531-tls-certs\") pod \"882dc67e-fe87-4033-8be0-bfdc48845531\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") "
Oct 29 22:56:35.696885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.696465    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-model-cache\") pod \"882dc67e-fe87-4033-8be0-bfdc48845531\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") "
Oct 29 22:56:35.696885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.696501    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-dshm\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-dshm\") pod \"882dc67e-fe87-4033-8be0-bfdc48845531\" (UID: \"882dc67e-fe87-4033-8be0-bfdc48845531\") "
Oct 29 22:56:35.696885 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.696664    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-tmp-dir" pod "882dc67e-fe87-4033-8be0-bfdc48845531" (UID: "882dc67e-fe87-4033-8be0-bfdc48845531"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:56:35.697386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.697304    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:56:35.697386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.697346    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:56:35.697386 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.697368    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-model-cache" pod "882dc67e-fe87-4033-8be0-bfdc48845531" (UID: "882dc67e-fe87-4033-8be0-bfdc48845531"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:56:35.700155 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.700115    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882dc67e-fe87-4033-8be0-bfdc48845531-kube-api-access-jcxnk" pod "882dc67e-fe87-4033-8be0-bfdc48845531" (UID: "882dc67e-fe87-4033-8be0-bfdc48845531"). InnerVolumeSpecName "kube-api-access-jcxnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:35.704558 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.704217    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/882dc67e-fe87-4033-8be0-bfdc48845531-tls-certs" pod "882dc67e-fe87-4033-8be0-bfdc48845531" (UID: "882dc67e-fe87-4033-8be0-bfdc48845531"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:56:35.704558 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.704497    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-dshm" pod "882dc67e-fe87-4033-8be0-bfdc48845531" (UID: "882dc67e-fe87-4033-8be0-bfdc48845531"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:56:35.798530 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.798497    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/882dc67e-fe87-4033-8be0-bfdc48845531-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:56:35.798530 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.798528    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:56:35.798530 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.798539    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/882dc67e-fe87-4033-8be0-bfdc48845531-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:56:35.798785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:35.798548    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcxnk\" (UniqueName: \"kubernetes.io/projected/882dc67e-fe87-4033-8be0-bfdc48845531-kube-api-access-jcxnk\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:56:36.310092 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:36.310059    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"
Oct 29 22:56:36.315900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:36.315870    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8" event={"ID":"882dc67e-fe87-4033-8be0-bfdc48845531","Type":"ContainerDied","Data":"cf220609dfd6fadfd6172d2622fd373f850c3e16be13821a47aeabddbf344e1a"}
Oct 29 22:56:36.316048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:36.315917    4559 scope.go:122] "RemoveContainer" containerID="201536958e6ab2e917cc2390560277bee542945baa4d703a34fc3f7e9d3b9a79"
Oct 29 22:56:36.377090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:36.377050    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"]
Oct 29 22:56:36.387299 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:36.387261    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-920032/auth-test-model-920032-kserve-6d7cd9d6df-zqkc8"]
Oct 29 22:56:37.230738 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:37.230705    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="882dc67e-fe87-4033-8be0-bfdc48845531" path="/var/lib/kubelet/pods/882dc67e-fe87-4033-8be0-bfdc48845531/volumes"
Oct 29 22:56:42.990378 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:42.990333    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"]
Oct 29 22:56:43.789125 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:56:43.789088    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:56:44.068474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.068380    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"]
Oct 29 22:56:44.069854 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.069794    4559 cpu_manager.go:414] "containerMap: removing container" podUID="882dc67e-fe87-4033-8be0-bfdc48845531" containerName="main"
Oct 29 22:56:44.069854 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.069856    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="882dc67e-fe87-4033-8be0-bfdc48845531" containerName="main"
Oct 29 22:56:44.070117 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.070098    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="882dc67e-fe87-4033-8be0-bfdc48845531" containerName="main"
Oct 29 22:56:44.077719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.077691    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.107746 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.107716    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b8f54c-openshift-default-dockercfg-2xsnx\""
Oct 29 22:56:44.108450 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.108419    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"]
Oct 29 22:56:44.217304 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.217263    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-data\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.217525 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.217409    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-workload-socket\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.217525 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.217456    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-envoy\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.217525 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.217516    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-credential-socket\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.217687 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.217656    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8gr4\" (UniqueName: \"kubernetes.io/projected/2c3db4b3-0ba7-49d3-9349-327aa6022a71-kube-api-access-w8gr4\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.217752 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.217719    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-token\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.217923 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.217905    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istiod-ca-cert\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.218005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.217943    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-podinfo\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.218085 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.218072    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-workload-certs\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.334911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.323301    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-workload-certs\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.334911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.323531    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-data\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.334911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.323627    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-workload-socket\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.334911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.323659    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-envoy\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.334911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.323727    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-credential-socket\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.334911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.323778    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8gr4\" (UniqueName: \"kubernetes.io/projected/2c3db4b3-0ba7-49d3-9349-327aa6022a71-kube-api-access-w8gr4\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.334911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.323859    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-token\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.334911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.324015    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istiod-ca-cert\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.334911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.324049    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-podinfo\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.334911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.327659    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-data\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.334911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.327956    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-workload-certs\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.334911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.328708    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-credential-socket\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.334911 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.329015    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-workload-socket\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.375836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.374551    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-podinfo\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.379833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.376660    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istiod-ca-cert\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.379833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.377130    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-envoy\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.398836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.393027    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8gr4\" (UniqueName: \"kubernetes.io/projected/2c3db4b3-0ba7-49d3-9349-327aa6022a71-kube-api-access-w8gr4\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.498908 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.498865    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:56:44.571666 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.571589    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-token\") pod \"e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") " pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:44.629850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:44.629389    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:45.038753 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:45.038719    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"]
Oct 29 22:56:45.041080 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:56:45.041034    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3db4b3_0ba7_49d3_9349_327aa6022a71.slice/crio-674c6621e68efbddc56794fa4120597f7bcc43cf4517eded5f76b1aca61271ac WatchSource:0}: Error finding container 674c6621e68efbddc56794fa4120597f7bcc43cf4517eded5f76b1aca61271ac: Status 404 returned error can't find the container with id 674c6621e68efbddc56794fa4120597f7bcc43cf4517eded5f76b1aca61271ac
Oct 29 22:56:45.044761 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:45.044721    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:56:45.044923 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:45.044829    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:56:45.044923 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:45.044869    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:56:45.479046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:45.478998    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:45.479594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:45.479065    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm" podUID="08448bc0-6fde-4460-bdf7-d65409faa732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:45.639239 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:45.639203    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt" event={"ID":"2c3db4b3-0ba7-49d3-9349-327aa6022a71","Type":"ContainerStarted","Data":"2213aca4d64dab8d98d60076dc33ffa470aba170197d61a8e08e169e526f13ff"}
Oct 29 22:56:45.639474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:45.639271    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt" event={"ID":"2c3db4b3-0ba7-49d3-9349-327aa6022a71","Type":"ContainerStarted","Data":"674c6621e68efbddc56794fa4120597f7bcc43cf4517eded5f76b1aca61271ac"}
Oct 29 22:56:45.685997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:45.685934    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt" podStartSLOduration=1.685918641 podStartE2EDuration="1.685918641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:45.680253266 +0000 UTC m=+2544.591696947" watchObservedRunningTime="2026-10-29 22:56:45.685918641 +0000 UTC m=+2544.597362194"
Oct 29 22:56:46.347006 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:46.346970    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"
Oct 29 22:56:46.348827 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:46.348780    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"
Oct 29 22:56:46.640993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:46.634259    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:46.646137 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:46.644134    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:46.651861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:46.651562    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:46.653338 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:46.653284    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:56:47.710915 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:47.703299    4559 generic.go:356] "Generic (PLEG): container finished" podID="08448bc0-6fde-4460-bdf7-d65409faa732" containerID="8096bfd682074ff9db2696aacd1b98126b100ed8cc05bf544a005060fb106b97" exitCode=0
Oct 29 22:56:47.710915 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:47.704500    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm" event={"ID":"08448bc0-6fde-4460-bdf7-d65409faa732","Type":"ContainerDied","Data":"8096bfd682074ff9db2696aacd1b98126b100ed8cc05bf544a005060fb106b97"}
Oct 29 22:56:47.710915 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:47.706650    4559 scope.go:122] "RemoveContainer" containerID="8096bfd682074ff9db2696aacd1b98126b100ed8cc05bf544a005060fb106b97"
Oct 29 22:56:47.715014 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:47.714967    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:56:47.715412 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:47.715370    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:56:47.715662 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:47.715622    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:56:48.713527 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:48.713489    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm" event={"ID":"08448bc0-6fde-4460-bdf7-d65409faa732","Type":"ContainerStarted","Data":"9b3cac06c5349688137a7896340b11dc9cd33a15536c8684df48bcf6422e7395"}
Oct 29 22:56:49.057900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:49.048764    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:56:49.057900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:49.048961    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 29 22:56:49.057900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:49.049004    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm" podUID="08448bc0-6fde-4460-bdf7-d65409faa732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 29 22:56:50.130834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:50.130251    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 29 22:56:50.130834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:50.130308    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm" podUID="08448bc0-6fde-4460-bdf7-d65409faa732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 29 22:56:50.436835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:50.436340    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"]
Oct 29 22:56:50.486093 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:50.486064    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"]
Oct 29 22:56:50.486537 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:50.486279    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"
Oct 29 22:56:50.501504 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:50.501477    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b8f54c\""
Oct 29 22:56:50.513835 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:50.513791    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b8f54c-dockercfg-glwcr\""
Oct 29 22:56:50.583775 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:50.583740    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ccb29b-516f-4caf-b86f-df0d49744ac1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw\" (UID: \"98ccb29b-516f-4caf-b86f-df0d49744ac1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"
Oct 29 22:56:50.584062 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:50.583979    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4lql\" (UniqueName: \"kubernetes.io/projected/98ccb29b-516f-4caf-b86f-df0d49744ac1-kube-api-access-j4lql\") pod \"payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw\" (UID: \"98ccb29b-516f-4caf-b86f-df0d49744ac1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"
Oct 29 22:56:50.685523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:50.685491    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4lql\" (UniqueName: \"kubernetes.io/projected/98ccb29b-516f-4caf-b86f-df0d49744ac1-kube-api-access-j4lql\") pod \"payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw\" (UID: \"98ccb29b-516f-4caf-b86f-df0d49744ac1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"
Oct 29 22:56:50.685703 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:50.685665    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ccb29b-516f-4caf-b86f-df0d49744ac1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw\" (UID: \"98ccb29b-516f-4caf-b86f-df0d49744ac1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"
Oct 29 22:56:50.686617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:50.686594    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ccb29b-516f-4caf-b86f-df0d49744ac1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw\" (UID: \"98ccb29b-516f-4caf-b86f-df0d49744ac1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"
Oct 29 22:56:50.705977 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:50.705902    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4lql\" (UniqueName: \"kubernetes.io/projected/98ccb29b-516f-4caf-b86f-df0d49744ac1-kube-api-access-j4lql\") pod \"payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw\" (UID: \"98ccb29b-516f-4caf-b86f-df0d49744ac1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"
Oct 29 22:56:50.803007 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:50.802968    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"
Oct 29 22:56:51.061023 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:51.060915    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 29 22:56:51.061023 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:51.060989    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm" podUID="08448bc0-6fde-4460-bdf7-d65409faa732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 29 22:56:51.290277 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:51.290197    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"]
Oct 29 22:56:51.773880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:51.773836    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw" event={"ID":"98ccb29b-516f-4caf-b86f-df0d49744ac1","Type":"ContainerStarted","Data":"3accc652d0c7f629cfe477d0f26ad3cb8ffde01d39e703a449633d338bced3a7"}
Oct 29 22:56:51.774104 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:51.773909    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw" event={"ID":"98ccb29b-516f-4caf-b86f-df0d49744ac1","Type":"ContainerStarted","Data":"9c3f1fdd7122ce9f38c9fd2d8f195d6cb60d56cd64ee38c8b0e9ebd10c6c733b"}
Oct 29 22:56:51.774104 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:51.773940    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"
Oct 29 22:56:51.815046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:51.814993    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw" podStartSLOduration=2.8149771919999997 podStartE2EDuration="2.814977192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:51.812628034 +0000 UTC m=+2550.724071590" watchObservedRunningTime="2026-10-29 22:56:51.814977192 +0000 UTC m=+2550.726420781"
Oct 29 22:56:52.056890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:52.053386    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 29 22:56:52.056890 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:52.053444    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm" podUID="08448bc0-6fde-4460-bdf7-d65409faa732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 29 22:56:53.068249 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:53.068208    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 29 22:56:53.068896 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:53.068862    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm" podUID="08448bc0-6fde-4460-bdf7-d65409faa732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 29 22:56:54.055282 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:54.055241    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 29 22:56:54.056495 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:54.056460    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm" podUID="08448bc0-6fde-4460-bdf7-d65409faa732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 29 22:56:55.053064 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:55.053028    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:56:55.053597 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:55.053423    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:56:55.054133 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:55.054112    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:56:58.019329 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.019291    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz" podUID="3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93" containerName="main" containerID="cri-o://b83d4441342378ff1d249b25e60fb230cd218efd9d57e6ac859c12abef535803" gracePeriod=45
Oct 29 22:56:58.342427 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.342388    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:56:58.509305 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.509273    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-kube-api-access-jphv7\" (UniqueName: \"kubernetes.io/projected/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-kube-api-access-jphv7\") pod \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") "
Oct 29 22:56:58.509534 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.509517    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-home\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-home\") pod \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") "
Oct 29 22:56:58.509597 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.509554    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-dshm\") pod \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") "
Oct 29 22:56:58.509669 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.509654    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-tls-certs\") pod \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") "
Oct 29 22:56:58.509716 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.509688    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-model-cache\") pod \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") "
Oct 29 22:56:58.509920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.509848    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-tmp-dir\") pod \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\" (UID: \"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93\") "
Oct 29 22:56:58.509920 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.509885    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-home" pod "3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93" (UID: "3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:56:58.510222 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.510198    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-tmp-dir" pod "3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93" (UID: "3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:56:58.510306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.510254    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-model-cache" pod "3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93" (UID: "3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:56:58.510657 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.510638    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:56:58.510750 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.510657    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:56:58.510750 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.510668    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:56:58.511933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.511912    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-kube-api-access-jphv7" pod "3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93" (UID: "3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93"). InnerVolumeSpecName "kube-api-access-jphv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:58.512090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.512072    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-tls-certs" pod "3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93" (UID: "3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:56:58.512140 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.512102    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-dshm" pod "3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93" (UID: "3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:56:58.612211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.612122    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:56:58.612211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.612152    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:56:58.612211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.612162    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jphv7\" (UniqueName: \"kubernetes.io/projected/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93-kube-api-access-jphv7\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:56:58.866798 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.866700    4559 generic.go:356] "Generic (PLEG): container finished" podID="3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93" containerID="b83d4441342378ff1d249b25e60fb230cd218efd9d57e6ac859c12abef535803" exitCode=0
Oct 29 22:56:58.866798 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.866742    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"
Oct 29 22:56:58.866798 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.866782    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz" event={"ID":"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93","Type":"ContainerDied","Data":"b83d4441342378ff1d249b25e60fb230cd218efd9d57e6ac859c12abef535803"}
Oct 29 22:56:58.867098 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.866864    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz" event={"ID":"3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93","Type":"ContainerDied","Data":"5da072a4b5ab2d883e3b9f2b4a235cb2074fef5c20d3476988db36463bd414c7"}
Oct 29 22:56:58.867098 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.866887    4559 scope.go:122] "RemoveContainer" containerID="b83d4441342378ff1d249b25e60fb230cd218efd9d57e6ac859c12abef535803"
Oct 29 22:56:58.885790 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.885770    4559 scope.go:122] "RemoveContainer" containerID="b83d4441342378ff1d249b25e60fb230cd218efd9d57e6ac859c12abef535803"
Oct 29 22:56:58.886143 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:56:58.886121    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b83d4441342378ff1d249b25e60fb230cd218efd9d57e6ac859c12abef535803\": container with ID starting with b83d4441342378ff1d249b25e60fb230cd218efd9d57e6ac859c12abef535803 not found: ID does not exist" containerID="b83d4441342378ff1d249b25e60fb230cd218efd9d57e6ac859c12abef535803"
Oct 29 22:56:58.886212 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.886154    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b83d4441342378ff1d249b25e60fb230cd218efd9d57e6ac859c12abef535803"} err="failed to get container status \"b83d4441342378ff1d249b25e60fb230cd218efd9d57e6ac859c12abef535803\": rpc error: code = NotFound desc = could not find container \"b83d4441342378ff1d249b25e60fb230cd218efd9d57e6ac859c12abef535803\": container with ID starting with b83d4441342378ff1d249b25e60fb230cd218efd9d57e6ac859c12abef535803 not found: ID does not exist"
Oct 29 22:56:58.922722 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.922687    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"]
Oct 29 22:56:58.929778 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:58.929745    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-02720a/auth-test-model-02720a-kserve-54bc7f774f-nz6fz"]
Oct 29 22:56:59.230917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:56:59.230887    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93" path="/var/lib/kubelet/pods/3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93/volumes"
Oct 29 22:57:02.799095 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:02.799019    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"
Oct 29 22:57:13.696595 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:57:13.696554    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:57:32.168728 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.168683    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"]
Oct 29 22:57:32.170169 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.170145    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93" containerName="main"
Oct 29 22:57:32.170265 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.170171    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93" containerName="main"
Oct 29 22:57:32.170380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.170368    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee1f4f6-4924-4dfd-8bab-af2c55dd4c93" containerName="main"
Oct 29 22:57:32.176859 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.176787    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.189606 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.189569    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"]
Oct 29 22:57:32.197352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.197322    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"]
Oct 29 22:57:32.210769 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.210719    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-istio-data\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.210975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.210839    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-istio-envoy\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.210975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.210870    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-workload-certs\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.211124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.210975    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec19a061-f45f-42f2-aea1-da03fe800597-istio-podinfo\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.211189 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.211140    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24g72\" (UniqueName: \"kubernetes.io/projected/ec19a061-f45f-42f2-aea1-da03fe800597-kube-api-access-24g72\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.211256 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.211211    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec19a061-f45f-42f2-aea1-da03fe800597-istiod-ca-cert\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.211317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.211261    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-workload-socket\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.211381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.211352    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec19a061-f45f-42f2-aea1-da03fe800597-istio-token\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.211440 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.211431    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-credential-socket\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.269344 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.269310    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-83f120-openshift-default-dockercfg-rmv78\""
Oct 29 22:57:32.313661 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.313625    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec19a061-f45f-42f2-aea1-da03fe800597-istio-token\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.317863 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.317462    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-credential-socket\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.319190 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.319163    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-istio-data\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.319562 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.319545    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-istio-envoy\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.319729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.319712    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-workload-certs\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.320439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.318472    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-credential-socket\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.320952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.320933    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-workload-certs\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.322024 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.321710    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-istio-data\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.322426 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.322407    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec19a061-f45f-42f2-aea1-da03fe800597-istio-podinfo\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.322611 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.322595    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24g72\" (UniqueName: \"kubernetes.io/projected/ec19a061-f45f-42f2-aea1-da03fe800597-kube-api-access-24g72\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.322759 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.322747    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec19a061-f45f-42f2-aea1-da03fe800597-istiod-ca-cert\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.322962 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.322947    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-workload-socket\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.323468 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.323448    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-workload-socket\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.324682 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.324656    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-istio-envoy\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.325486 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.325462    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec19a061-f45f-42f2-aea1-da03fe800597-istiod-ca-cert\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.346615 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.346580    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec19a061-f45f-42f2-aea1-da03fe800597-istio-podinfo\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.352513 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.352474    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24g72\" (UniqueName: \"kubernetes.io/projected/ec19a061-f45f-42f2-aea1-da03fe800597-kube-api-access-24g72\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.352957 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.352908    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec19a061-f45f-42f2-aea1-da03fe800597-istio-token\") pod \"e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") " pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:32.510692 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:32.510598    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:33.596550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:33.596516    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"]
Oct 29 22:57:33.597143 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:57:33.597118    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec19a061_f45f_42f2_aea1_da03fe800597.slice/crio-d8a3e0b49593ee777b07c3122cb572606479932e87bfeba68e35607680f8471d WatchSource:0}: Error finding container d8a3e0b49593ee777b07c3122cb572606479932e87bfeba68e35607680f8471d: Status 404 returned error can't find the container with id d8a3e0b49593ee777b07c3122cb572606479932e87bfeba68e35607680f8471d
Oct 29 22:57:33.600900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:33.600860    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:57:33.600989 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:33.600961    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:57:33.601029 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:33.601004    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:57:34.329438 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:34.329396    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r" event={"ID":"ec19a061-f45f-42f2-aea1-da03fe800597","Type":"ContainerStarted","Data":"305c5988baf041f6f2d0a220e8df32aec6982d2b2cbd1fce10400f2ed240bebb"}
Oct 29 22:57:34.329645 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:34.329468    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r" event={"ID":"ec19a061-f45f-42f2-aea1-da03fe800597","Type":"ContainerStarted","Data":"d8a3e0b49593ee777b07c3122cb572606479932e87bfeba68e35607680f8471d"}
Oct 29 22:57:34.395580 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:34.395523    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r" podStartSLOduration=2.395498915 podStartE2EDuration="2.395498915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:34.380951863 +0000 UTC m=+2593.292395417" watchObservedRunningTime="2026-10-29 22:57:34.395498915 +0000 UTC m=+2593.306942470"
Oct 29 22:57:34.511599 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:34.511562    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:34.517829 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:34.517780    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:35.338409 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:35.338378    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:35.339614 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:35.339474    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:57:37.190925 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:37.190876    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b" podUID="6594f54e-cae8-43df-a4c7-8096eaee4a70" containerName="payload-pre-processing" containerID="cri-o://ead5f990bb684821ef3a657936c4db8bbcd1f0be95482394fbd8a55bb4362f20" gracePeriod=25
Oct 29 22:57:37.423999 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:37.423920    4559 generic.go:356] "Generic (PLEG): container finished" podID="6594f54e-cae8-43df-a4c7-8096eaee4a70" containerID="ead5f990bb684821ef3a657936c4db8bbcd1f0be95482394fbd8a55bb4362f20" exitCode=0
Oct 29 22:57:37.430846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:37.425245    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b" event={"ID":"6594f54e-cae8-43df-a4c7-8096eaee4a70","Type":"ContainerDied","Data":"ead5f990bb684821ef3a657936c4db8bbcd1f0be95482394fbd8a55bb4362f20"}
Oct 29 22:57:37.603189 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:57:37.603147    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6594f54e_cae8_43df_a4c7_8096eaee4a70.slice/crio-ead5f990bb684821ef3a657936c4db8bbcd1f0be95482394fbd8a55bb4362f20.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6594f54e_cae8_43df_a4c7_8096eaee4a70.slice/crio-conmon-ead5f990bb684821ef3a657936c4db8bbcd1f0be95482394fbd8a55bb4362f20.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:57:37.902002 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:37.901934    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"
Oct 29 22:57:38.048736 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:38.048702    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6594f54e-cae8-43df-a4c7-8096eaee4a70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6594f54e-cae8-43df-a4c7-8096eaee4a70-plugins-config-volume\") pod \"6594f54e-cae8-43df-a4c7-8096eaee4a70\" (UID: \"6594f54e-cae8-43df-a4c7-8096eaee4a70\") "
Oct 29 22:57:38.048970 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:38.048789    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6594f54e-cae8-43df-a4c7-8096eaee4a70-kube-api-access-lhps6\" (UniqueName: \"kubernetes.io/projected/6594f54e-cae8-43df-a4c7-8096eaee4a70-kube-api-access-lhps6\") pod \"6594f54e-cae8-43df-a4c7-8096eaee4a70\" (UID: \"6594f54e-cae8-43df-a4c7-8096eaee4a70\") "
Oct 29 22:57:38.049167 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:38.049141    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6594f54e-cae8-43df-a4c7-8096eaee4a70-plugins-config-volume" pod "6594f54e-cae8-43df-a4c7-8096eaee4a70" (UID: "6594f54e-cae8-43df-a4c7-8096eaee4a70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:38.049635 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:38.049610    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6594f54e-cae8-43df-a4c7-8096eaee4a70-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:57:38.051477 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:38.051416    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6594f54e-cae8-43df-a4c7-8096eaee4a70-kube-api-access-lhps6" pod "6594f54e-cae8-43df-a4c7-8096eaee4a70" (UID: "6594f54e-cae8-43df-a4c7-8096eaee4a70"). InnerVolumeSpecName "kube-api-access-lhps6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:38.153839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:38.153593    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhps6\" (UniqueName: \"kubernetes.io/projected/6594f54e-cae8-43df-a4c7-8096eaee4a70-kube-api-access-lhps6\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:57:38.437952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:38.437922    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"
Oct 29 22:57:38.437952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:38.437932    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b" event={"ID":"6594f54e-cae8-43df-a4c7-8096eaee4a70","Type":"ContainerDied","Data":"e4547023db436f5a29cd6c0a4209455f4216ed1ea869acc2aad0a6e88074e5c7"}
Oct 29 22:57:38.438544 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:38.438009    4559 scope.go:122] "RemoveContainer" containerID="ead5f990bb684821ef3a657936c4db8bbcd1f0be95482394fbd8a55bb4362f20"
Oct 29 22:57:38.520946 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:38.520903    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"]
Oct 29 22:57:38.534134 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:38.534099    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-d6c677c75-9zj9b"]
Oct 29 22:57:39.304845 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:39.295341    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6594f54e-cae8-43df-a4c7-8096eaee4a70" path="/var/lib/kubelet/pods/6594f54e-cae8-43df-a4c7-8096eaee4a70/volumes"
Oct 29 22:57:40.596903 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:40.596574    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:57:40.603397 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:40.602554    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.677837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:40.675911    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.681837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:40.678352    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:57:41.292853 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:41.289987    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.293205 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:41.293161    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.336309 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:41.335028    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.606902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:41.599782    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:57:41.606902 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:41.600357    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:57:41.693838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:41.682658    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.694883 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:41.694854    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:57:41.696824 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:41.696779    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:57:41.697634 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:41.697615    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:57:42.372833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:42.307922    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:42.372833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:42.308238    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:57:42.372833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:42.308729    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:57:42.372833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:42.308789    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:57:42.372833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:42.355841    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:42.372833 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:42.356318    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:57:42.457837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:42.449508    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 22:57:42.462834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:42.458155    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 22:57:42.536729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:42.536695    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 22:57:42.555996 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:42.552444    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 22:57:42.600412 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:42.600238    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:57:42.724894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:42.705855    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:42.821007 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:42.820977    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:57:42.893472 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:42.893442    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 22:57:43.697053 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:57:43.697015    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:57:52.879068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:52.878967    4559 generic.go:356] "Generic (PLEG): container finished" podID="74383eae-9fc4-44f4-9279-a2c085c4eeb9" containerID="c3a133d3083c54ae82895401079bdcdd5a7276abbfec38ddcccc7ff872dd37d1" exitCode=0
Oct 29 22:57:52.879068 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:52.879044    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm" event={"ID":"74383eae-9fc4-44f4-9279-a2c085c4eeb9","Type":"ContainerDied","Data":"c3a133d3083c54ae82895401079bdcdd5a7276abbfec38ddcccc7ff872dd37d1"}
Oct 29 22:57:52.879649 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:52.879629    4559 scope.go:122] "RemoveContainer" containerID="c3a133d3083c54ae82895401079bdcdd5a7276abbfec38ddcccc7ff872dd37d1"
Oct 29 22:57:52.881778 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:52.881750    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:57:52.881897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:52.881850    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:57:52.881897 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:52.881885    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:57:52.943241 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:52.943209    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:57:53.890547 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:53.890508    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm" event={"ID":"74383eae-9fc4-44f4-9279-a2c085c4eeb9","Type":"ContainerStarted","Data":"62bf2da83f4d8881350b257564586f8198cd717364eed97a85a80567f5f8a6a7"}
Oct 29 22:57:54.385972 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:54.385862    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:57:54.387362 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:54.387333    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 29 22:57:54.387537 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:54.387394    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm" podUID="74383eae-9fc4-44f4-9279-a2c085c4eeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 29 22:57:55.385720 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.385676    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 29 22:57:55.386207 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.385749    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm" podUID="74383eae-9fc4-44f4-9279-a2c085c4eeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 29 22:57:55.624352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.624316    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"]
Oct 29 22:57:55.625339 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.625314    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6594f54e-cae8-43df-a4c7-8096eaee4a70" containerName="payload-pre-processing"
Oct 29 22:57:55.625339 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.625338    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6594f54e-cae8-43df-a4c7-8096eaee4a70" containerName="payload-pre-processing"
Oct 29 22:57:55.625547 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.625532    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6594f54e-cae8-43df-a4c7-8096eaee4a70" containerName="payload-pre-processing"
Oct 29 22:57:55.629877 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.629856    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.682739 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.681954    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"]
Oct 29 22:57:55.768837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.739063    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-podinfo\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.768837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.739121    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-envoy\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.768837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.739297    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cccl\" (UniqueName: \"kubernetes.io/projected/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-kube-api-access-7cccl\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.768837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.739421    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-credential-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.768837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.739463    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-workload-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.768837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.739553    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istiod-ca-cert\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.768837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.739606    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-data\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.768837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.739708    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-token\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.768837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.739861    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-workload-certs\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.858837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.857030    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-workload-certs\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.858837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.857131    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-podinfo\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.858837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.857165    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-envoy\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.858837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.857308    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cccl\" (UniqueName: \"kubernetes.io/projected/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-kube-api-access-7cccl\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.858837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.857404    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-credential-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.858837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.857444    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-workload-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.858837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.857511    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istiod-ca-cert\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.858837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.857556    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-data\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.858837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.857645    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-token\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.858837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.858537    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-workload-certs\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.863630 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.863597    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-credential-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.866217 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.865152    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istiod-ca-cert\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.866217 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.865510    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-workload-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.866217 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.865525    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-data\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.868672 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.868431    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-podinfo\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.870262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.870239    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-envoy\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.877178 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.877148    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-token\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.883953 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.883923    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cccl\" (UniqueName: \"kubernetes.io/projected/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-kube-api-access-7cccl\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:55.992690 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:55.992564    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:56.368589 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:56.368538    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"]
Oct 29 22:57:56.372878 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:57:56.372798    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a55532b_2fce_4bce_9b7b_9ef09bd7aa3d.slice/crio-0ce837734b8249bcf7fa631241023d65820e2b652aa4d139986cd5fbfaa6a961 WatchSource:0}: Error finding container 0ce837734b8249bcf7fa631241023d65820e2b652aa4d139986cd5fbfaa6a961: Status 404 returned error can't find the container with id 0ce837734b8249bcf7fa631241023d65820e2b652aa4d139986cd5fbfaa6a961
Oct 29 22:57:56.377518 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:56.377479    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:57:56.377642 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:56.377563    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:57:56.377642 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:56.377594    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:57:56.385311 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:56.385278    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 29 22:57:56.385431 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:56.385339    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm" podUID="74383eae-9fc4-44f4-9279-a2c085c4eeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 29 22:57:56.986133 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:56.982092    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt" event={"ID":"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d","Type":"ContainerStarted","Data":"acec9394fbd0aa8edb2cd77af8478809f0c45ad81c609adc8e3e9d20c688c938"}
Oct 29 22:57:56.986133 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:56.982167    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt" event={"ID":"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d","Type":"ContainerStarted","Data":"0ce837734b8249bcf7fa631241023d65820e2b652aa4d139986cd5fbfaa6a961"}
Oct 29 22:57:56.993969 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:56.993938    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:57:57.032306 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:57.032240    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt" podStartSLOduration=2.032225026 podStartE2EDuration="2.032225026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:57.027153798 +0000 UTC m=+2615.938597380" watchObservedRunningTime="2026-10-29 22:57:57.032225026 +0000 UTC m=+2615.943668574"
Oct 29 22:57:57.385494 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:57.385267    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 29 22:57:57.385494 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:57.385351    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm" podUID="74383eae-9fc4-44f4-9279-a2c085c4eeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 29 22:57:57.995241 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:57.995201    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 29 22:57:57.995927 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:57.995886    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt" podUID="2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 29 22:57:58.317682 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:58.316149    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"]
Oct 29 22:57:58.317682 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:58.316392    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t" podUID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" containerName="istio-proxy" containerID="cri-o://47d0a75cc39ffcafd7635f4c3e1e340478424350790f9a3d14c1d615c83c3ef1" gracePeriod=30
Oct 29 22:57:58.405837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:58.400779    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 29 22:57:58.405837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:58.400861    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm" podUID="74383eae-9fc4-44f4-9279-a2c085c4eeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 29 22:57:59.029828 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:59.026448    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 29 22:57:59.029828 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:59.026506    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt" podUID="2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 29 22:57:59.394839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:59.391990    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 29 22:57:59.394839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:57:59.392087    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm" podUID="74383eae-9fc4-44f4-9279-a2c085c4eeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 29 22:58:00.022843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:00.020968    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 29 22:58:00.022843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:00.021034    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt" podUID="2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 29 22:58:00.390303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:00.390275    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:58:00.390733 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:00.390706    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:58:00.391402 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:00.391385    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 22:58:00.994154 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:00.994121    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 29 22:58:00.994353 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:00.994176    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt" podUID="2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 29 22:58:01.871211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:01.871170    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:58:01.871784 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:01.871753    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t" podUID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:58:01.993689 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:01.993651    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 29 22:58:01.993937 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:01.993708    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt" podUID="2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 29 22:58:02.993448 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:02.993415    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 29 22:58:02.993931 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:02.993470    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt" podUID="2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 29 22:58:03.814837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:03.814048    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:58:03.919838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:03.910395    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-podinfo\") pod \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") "
Oct 29 22:58:03.919838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:03.910476    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-data\") pod \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") "
Oct 29 22:58:03.919838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:03.910572    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-kube-api-access-vhgnc\" (UniqueName: \"kubernetes.io/projected/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-kube-api-access-vhgnc\") pod \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") "
Oct 29 22:58:03.919838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:03.910683    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-credential-socket\") pod \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") "
Oct 29 22:58:03.919838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:03.910789    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istiod-ca-cert\") pod \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") "
Oct 29 22:58:03.919838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:03.910879    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-envoy\") pod \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") "
Oct 29 22:58:03.919838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:03.911022    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-workload-socket\") pod \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") "
Oct 29 22:58:03.919838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:03.911106    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-token\" (UniqueName: \"kubernetes.io/projected/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-token\") pod \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") "
Oct 29 22:58:03.919838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:03.911143    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-workload-certs\") pod \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\" (UID: \"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2\") "
Oct 29 22:58:03.919838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:03.913093    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-credential-socket" pod "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" (UID: "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:03.919838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:03.913962    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-workload-socket" pod "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" (UID: "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:03.919838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:03.916237    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-data" pod "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" (UID: "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:03.919838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:03.918971    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-podinfo" pod "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" (UID: "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:58:03.920933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:03.920867    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-workload-certs" pod "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" (UID: "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:03.929834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:03.926008    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istiod-ca-cert" pod "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" (UID: "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:03.957515 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:03.957035    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-token" pod "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" (UID: "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:04.026688 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.011382    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 29 22:58:04.026688 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.011442    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt" podUID="2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 29 22:58:04.026688 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.014433    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:04.026688 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.014465    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:04.026688 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.014480    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:04.026688 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.014497    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:04.026688 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.014509    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:04.026688 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.014527    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:04.026688 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.014542    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:04.048248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.034050    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-envoy" pod "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" (UID: "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:04.048248 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.036686    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-kube-api-access-vhgnc" pod "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" (UID: "616f48d5-a65e-48ae-a0e4-b2d3f61d72d2"). InnerVolumeSpecName "kube-api-access-vhgnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:04.231894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.225154    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhgnc\" (UniqueName: \"kubernetes.io/projected/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-kube-api-access-vhgnc\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:04.231894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.225192    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:04.265888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.264646    4559 generic.go:356] "Generic (PLEG): container finished" podID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" containerID="47d0a75cc39ffcafd7635f4c3e1e340478424350790f9a3d14c1d615c83c3ef1" exitCode=0
Oct 29 22:58:04.265888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.264786    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t" event={"ID":"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2","Type":"ContainerDied","Data":"47d0a75cc39ffcafd7635f4c3e1e340478424350790f9a3d14c1d615c83c3ef1"}
Oct 29 22:58:04.265888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.264903    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t" event={"ID":"616f48d5-a65e-48ae-a0e4-b2d3f61d72d2","Type":"ContainerDied","Data":"a016d05dfc43702c7e58b479da42cbf635abdb2dfb0d78140b780979505820c1"}
Oct 29 22:58:04.265888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.264954    4559 scope.go:122] "RemoveContainer" containerID="47d0a75cc39ffcafd7635f4c3e1e340478424350790f9a3d14c1d615c83c3ef1"
Oct 29 22:58:04.265888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.265181    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"
Oct 29 22:58:04.294892 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.294865    4559 scope.go:122] "RemoveContainer" containerID="c6dcb7da1747cd025dc7d1a33d3277807b905d2546f0193e501f937c13241070"
Oct 29 22:58:04.315131 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.315096    4559 scope.go:122] "RemoveContainer" containerID="47d0a75cc39ffcafd7635f4c3e1e340478424350790f9a3d14c1d615c83c3ef1"
Oct 29 22:58:04.315608 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:58:04.315578    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d0a75cc39ffcafd7635f4c3e1e340478424350790f9a3d14c1d615c83c3ef1\": container with ID starting with 47d0a75cc39ffcafd7635f4c3e1e340478424350790f9a3d14c1d615c83c3ef1 not found: ID does not exist" containerID="47d0a75cc39ffcafd7635f4c3e1e340478424350790f9a3d14c1d615c83c3ef1"
Oct 29 22:58:04.315738 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.315620    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d0a75cc39ffcafd7635f4c3e1e340478424350790f9a3d14c1d615c83c3ef1"} err="failed to get container status \"47d0a75cc39ffcafd7635f4c3e1e340478424350790f9a3d14c1d615c83c3ef1\": rpc error: code = NotFound desc = could not find container \"47d0a75cc39ffcafd7635f4c3e1e340478424350790f9a3d14c1d615c83c3ef1\": container with ID starting with 47d0a75cc39ffcafd7635f4c3e1e340478424350790f9a3d14c1d615c83c3ef1 not found: ID does not exist"
Oct 29 22:58:04.315738 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.315647    4559 scope.go:122] "RemoveContainer" containerID="c6dcb7da1747cd025dc7d1a33d3277807b905d2546f0193e501f937c13241070"
Oct 29 22:58:04.316374 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:58:04.316082    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6dcb7da1747cd025dc7d1a33d3277807b905d2546f0193e501f937c13241070\": container with ID starting with c6dcb7da1747cd025dc7d1a33d3277807b905d2546f0193e501f937c13241070 not found: ID does not exist" containerID="c6dcb7da1747cd025dc7d1a33d3277807b905d2546f0193e501f937c13241070"
Oct 29 22:58:04.316374 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.316117    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6dcb7da1747cd025dc7d1a33d3277807b905d2546f0193e501f937c13241070"} err="failed to get container status \"c6dcb7da1747cd025dc7d1a33d3277807b905d2546f0193e501f937c13241070\": rpc error: code = NotFound desc = could not find container \"c6dcb7da1747cd025dc7d1a33d3277807b905d2546f0193e501f937c13241070\": container with ID starting with c6dcb7da1747cd025dc7d1a33d3277807b905d2546f0193e501f937c13241070 not found: ID does not exist"
Oct 29 22:58:04.357694 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.357631    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"]
Oct 29 22:58:04.367742 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.367709    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c87bf1-openshift-default-648d4845d9-qx65t"]
Oct 29 22:58:04.998976 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.998945    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:58:04.999384 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:04.999364    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:58:05.000269 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:05.000247    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 22:58:05.231663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:05.231628    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" path="/var/lib/kubelet/pods/616f48d5-a65e-48ae-a0e4-b2d3f61d72d2/volumes"
Oct 29 22:58:10.648151 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:10.648105    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"]
Oct 29 22:58:10.648679 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:10.648403    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm" podUID="1c86b737-9464-4a60-9934-64096680f5f3" containerName="maas-api" containerID="cri-o://2f95d835c008bc3bd230bd6b0be4045c5f641409f2d731ec368d753b7c32d858" gracePeriod=30
Oct 29 22:58:10.797717 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:10.797671    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"]
Oct 29 22:58:10.798067 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:10.798011    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf" podUID="4875d982-49ff-4402-86c3-e7a9f97ffba7" containerName="payload-processing" containerID="cri-o://bc0a99739e8bc6cae10def6163e916be9ed593721a2d4df66ad707a65eb7195f" gracePeriod=30
Oct 29 22:58:10.827780 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:10.826303    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"]
Oct 29 22:58:10.827780 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:10.826559    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f" podUID="08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3" containerName="payload-pre-processing" containerID="cri-o://ab00e83b955d42298684de19e585846733b898aefb1c53b77950f87f910658a7" gracePeriod=30
Oct 29 22:58:11.107567 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.107168    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:58:11.203133 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.203041    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c86b737-9464-4a60-9934-64096680f5f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c86b737-9464-4a60-9934-64096680f5f3-openshift-service-ca\") pod \"1c86b737-9464-4a60-9934-64096680f5f3\" (UID: \"1c86b737-9464-4a60-9934-64096680f5f3\") "
Oct 29 22:58:11.203319 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.203135    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c86b737-9464-4a60-9934-64096680f5f3-kube-api-access-w7fjk\" (UniqueName: \"kubernetes.io/projected/1c86b737-9464-4a60-9934-64096680f5f3-kube-api-access-w7fjk\") pod \"1c86b737-9464-4a60-9934-64096680f5f3\" (UID: \"1c86b737-9464-4a60-9934-64096680f5f3\") "
Oct 29 22:58:11.203319 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.203223    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c86b737-9464-4a60-9934-64096680f5f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c86b737-9464-4a60-9934-64096680f5f3-maas-api-tls\") pod \"1c86b737-9464-4a60-9934-64096680f5f3\" (UID: \"1c86b737-9464-4a60-9934-64096680f5f3\") "
Oct 29 22:58:11.203458 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.203357    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c86b737-9464-4a60-9934-64096680f5f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c86b737-9464-4a60-9934-64096680f5f3-metrics-certs\") pod \"1c86b737-9464-4a60-9934-64096680f5f3\" (UID: \"1c86b737-9464-4a60-9934-64096680f5f3\") "
Oct 29 22:58:11.203458 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.203430    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c86b737-9464-4a60-9934-64096680f5f3-openshift-service-ca" pod "1c86b737-9464-4a60-9934-64096680f5f3" (UID: "1c86b737-9464-4a60-9934-64096680f5f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:11.204208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.204180    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c86b737-9464-4a60-9934-64096680f5f3-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:11.215189 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.215158    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c86b737-9464-4a60-9934-64096680f5f3-maas-api-tls" pod "1c86b737-9464-4a60-9934-64096680f5f3" (UID: "1c86b737-9464-4a60-9934-64096680f5f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:11.215361 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.215341    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c86b737-9464-4a60-9934-64096680f5f3-kube-api-access-w7fjk" pod "1c86b737-9464-4a60-9934-64096680f5f3" (UID: "1c86b737-9464-4a60-9934-64096680f5f3"). InnerVolumeSpecName "kube-api-access-w7fjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:11.218628 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.217574    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c86b737-9464-4a60-9934-64096680f5f3-metrics-certs" pod "1c86b737-9464-4a60-9934-64096680f5f3" (UID: "1c86b737-9464-4a60-9934-64096680f5f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:11.288543 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.288513    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"
Oct 29 22:58:11.308609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.307979    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c86b737-9464-4a60-9934-64096680f5f3-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:11.308949 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.308925    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7fjk\" (UniqueName: \"kubernetes.io/projected/1c86b737-9464-4a60-9934-64096680f5f3-kube-api-access-w7fjk\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:11.309143 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.309128    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c86b737-9464-4a60-9934-64096680f5f3-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:11.332113 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.332081    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"
Oct 29 22:58:11.338965 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.338929    4559 generic.go:356] "Generic (PLEG): container finished" podID="1c86b737-9464-4a60-9934-64096680f5f3" containerID="2f95d835c008bc3bd230bd6b0be4045c5f641409f2d731ec368d753b7c32d858" exitCode=0
Oct 29 22:58:11.339145 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.339113    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm" event={"ID":"1c86b737-9464-4a60-9934-64096680f5f3","Type":"ContainerDied","Data":"2f95d835c008bc3bd230bd6b0be4045c5f641409f2d731ec368d753b7c32d858"}
Oct 29 22:58:11.339222 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.339164    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm" event={"ID":"1c86b737-9464-4a60-9934-64096680f5f3","Type":"ContainerDied","Data":"898a1af21f3fffb2182c2e07b4d2a0a235b6645d5b9c0487ba6cf8a99bfc8d88"}
Oct 29 22:58:11.339222 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.339192    4559 scope.go:122] "RemoveContainer" containerID="2f95d835c008bc3bd230bd6b0be4045c5f641409f2d731ec368d753b7c32d858"
Oct 29 22:58:11.339363 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.339349    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"
Oct 29 22:58:11.342296 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.342271    4559 generic.go:356] "Generic (PLEG): container finished" podID="4875d982-49ff-4402-86c3-e7a9f97ffba7" containerID="bc0a99739e8bc6cae10def6163e916be9ed593721a2d4df66ad707a65eb7195f" exitCode=0
Oct 29 22:58:11.342755 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.342728    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf" event={"ID":"4875d982-49ff-4402-86c3-e7a9f97ffba7","Type":"ContainerDied","Data":"bc0a99739e8bc6cae10def6163e916be9ed593721a2d4df66ad707a65eb7195f"}
Oct 29 22:58:11.342970 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.342949    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf" event={"ID":"4875d982-49ff-4402-86c3-e7a9f97ffba7","Type":"ContainerDied","Data":"90506a35e50840b2b3950bb00bed4cc82a030d3500e776f69713bcac355c304a"}
Oct 29 22:58:11.343136 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.343122    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"
Oct 29 22:58:11.345071 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.345050    4559 generic.go:356] "Generic (PLEG): container finished" podID="08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3" containerID="ab00e83b955d42298684de19e585846733b898aefb1c53b77950f87f910658a7" exitCode=0
Oct 29 22:58:11.345282 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.345265    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f" event={"ID":"08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3","Type":"ContainerDied","Data":"ab00e83b955d42298684de19e585846733b898aefb1c53b77950f87f910658a7"}
Oct 29 22:58:11.345409 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.345396    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f" event={"ID":"08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3","Type":"ContainerDied","Data":"b2d1cf73126dcc5d96c14d7a4687d0f08d095bf7e7dcccf369bf4f7380b520dc"}
Oct 29 22:58:11.345657 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.345642    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"
Oct 29 22:58:11.365447 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.365415    4559 scope.go:122] "RemoveContainer" containerID="2f95d835c008bc3bd230bd6b0be4045c5f641409f2d731ec368d753b7c32d858"
Oct 29 22:58:11.365870 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:58:11.365840    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f95d835c008bc3bd230bd6b0be4045c5f641409f2d731ec368d753b7c32d858\": container with ID starting with 2f95d835c008bc3bd230bd6b0be4045c5f641409f2d731ec368d753b7c32d858 not found: ID does not exist" containerID="2f95d835c008bc3bd230bd6b0be4045c5f641409f2d731ec368d753b7c32d858"
Oct 29 22:58:11.365977 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.365883    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f95d835c008bc3bd230bd6b0be4045c5f641409f2d731ec368d753b7c32d858"} err="failed to get container status \"2f95d835c008bc3bd230bd6b0be4045c5f641409f2d731ec368d753b7c32d858\": rpc error: code = NotFound desc = could not find container \"2f95d835c008bc3bd230bd6b0be4045c5f641409f2d731ec368d753b7c32d858\": container with ID starting with 2f95d835c008bc3bd230bd6b0be4045c5f641409f2d731ec368d753b7c32d858 not found: ID does not exist"
Oct 29 22:58:11.365977 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.365911    4559 scope.go:122] "RemoveContainer" containerID="bc0a99739e8bc6cae10def6163e916be9ed593721a2d4df66ad707a65eb7195f"
Oct 29 22:58:11.382757 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.382501    4559 scope.go:122] "RemoveContainer" containerID="bc0a99739e8bc6cae10def6163e916be9ed593721a2d4df66ad707a65eb7195f"
Oct 29 22:58:11.383321 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:58:11.383012    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0a99739e8bc6cae10def6163e916be9ed593721a2d4df66ad707a65eb7195f\": container with ID starting with bc0a99739e8bc6cae10def6163e916be9ed593721a2d4df66ad707a65eb7195f not found: ID does not exist" containerID="bc0a99739e8bc6cae10def6163e916be9ed593721a2d4df66ad707a65eb7195f"
Oct 29 22:58:11.383321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.383048    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0a99739e8bc6cae10def6163e916be9ed593721a2d4df66ad707a65eb7195f"} err="failed to get container status \"bc0a99739e8bc6cae10def6163e916be9ed593721a2d4df66ad707a65eb7195f\": rpc error: code = NotFound desc = could not find container \"bc0a99739e8bc6cae10def6163e916be9ed593721a2d4df66ad707a65eb7195f\": container with ID starting with bc0a99739e8bc6cae10def6163e916be9ed593721a2d4df66ad707a65eb7195f not found: ID does not exist"
Oct 29 22:58:11.383321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.383072    4559 scope.go:122] "RemoveContainer" containerID="ab00e83b955d42298684de19e585846733b898aefb1c53b77950f87f910658a7"
Oct 29 22:58:11.404104 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.404080    4559 scope.go:122] "RemoveContainer" containerID="ab00e83b955d42298684de19e585846733b898aefb1c53b77950f87f910658a7"
Oct 29 22:58:11.404545 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:58:11.404523    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab00e83b955d42298684de19e585846733b898aefb1c53b77950f87f910658a7\": container with ID starting with ab00e83b955d42298684de19e585846733b898aefb1c53b77950f87f910658a7 not found: ID does not exist" containerID="ab00e83b955d42298684de19e585846733b898aefb1c53b77950f87f910658a7"
Oct 29 22:58:11.404651 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.404558    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab00e83b955d42298684de19e585846733b898aefb1c53b77950f87f910658a7"} err="failed to get container status \"ab00e83b955d42298684de19e585846733b898aefb1c53b77950f87f910658a7\": rpc error: code = NotFound desc = could not find container \"ab00e83b955d42298684de19e585846733b898aefb1c53b77950f87f910658a7\": container with ID starting with ab00e83b955d42298684de19e585846733b898aefb1c53b77950f87f910658a7 not found: ID does not exist"
Oct 29 22:58:11.410056 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.410021    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4875d982-49ff-4402-86c3-e7a9f97ffba7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4875d982-49ff-4402-86c3-e7a9f97ffba7-plugins-config-volume\") pod \"4875d982-49ff-4402-86c3-e7a9f97ffba7\" (UID: \"4875d982-49ff-4402-86c3-e7a9f97ffba7\") "
Oct 29 22:58:11.410193 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.410081    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3-plugins-config-volume\") pod \"08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3\" (UID: \"08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3\") "
Oct 29 22:58:11.410193 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.410156    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4875d982-49ff-4402-86c3-e7a9f97ffba7-kube-api-access-vc2mp\" (UniqueName: \"kubernetes.io/projected/4875d982-49ff-4402-86c3-e7a9f97ffba7-kube-api-access-vc2mp\") pod \"4875d982-49ff-4402-86c3-e7a9f97ffba7\" (UID: \"4875d982-49ff-4402-86c3-e7a9f97ffba7\") "
Oct 29 22:58:11.410315 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.410226    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3-kube-api-access-cqjwk\" (UniqueName: \"kubernetes.io/projected/08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3-kube-api-access-cqjwk\") pod \"08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3\" (UID: \"08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3\") "
Oct 29 22:58:11.411069 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.411035    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3-plugins-config-volume" pod "08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3" (UID: "08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:11.411566 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.411512    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4875d982-49ff-4402-86c3-e7a9f97ffba7-plugins-config-volume" pod "4875d982-49ff-4402-86c3-e7a9f97ffba7" (UID: "4875d982-49ff-4402-86c3-e7a9f97ffba7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:11.413226 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.413204    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3-kube-api-access-cqjwk" pod "08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3" (UID: "08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3"). InnerVolumeSpecName "kube-api-access-cqjwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:11.413664 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.413648    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4875d982-49ff-4402-86c3-e7a9f97ffba7-kube-api-access-vc2mp" pod "4875d982-49ff-4402-86c3-e7a9f97ffba7" (UID: "4875d982-49ff-4402-86c3-e7a9f97ffba7"). InnerVolumeSpecName "kube-api-access-vc2mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:11.457241 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.457103    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"]
Oct 29 22:58:11.464729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.464664    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c4b02-6cc87f44d4-44nnm"]
Oct 29 22:58:11.512719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.512678    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4875d982-49ff-4402-86c3-e7a9f97ffba7-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:11.512719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.512715    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:11.512719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.512728    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc2mp\" (UniqueName: \"kubernetes.io/projected/4875d982-49ff-4402-86c3-e7a9f97ffba7-kube-api-access-vc2mp\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:11.513011 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.512740    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqjwk\" (UniqueName: \"kubernetes.io/projected/08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3-kube-api-access-cqjwk\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:11.983198 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:11.983144    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"]
Oct 29 22:58:12.001320 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:12.001265    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-68c646b4dd-rx4cf"]
Oct 29 22:58:12.052964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:12.052916    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"]
Oct 29 22:58:12.054826 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:12.054782    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-c7cf79579-lj49f"]
Oct 29 22:58:13.231070 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:13.231034    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3" path="/var/lib/kubelet/pods/08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3/volumes"
Oct 29 22:58:13.231651 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:13.231631    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c86b737-9464-4a60-9934-64096680f5f3" path="/var/lib/kubelet/pods/1c86b737-9464-4a60-9934-64096680f5f3/volumes"
Oct 29 22:58:13.232162 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:13.232145    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4875d982-49ff-4402-86c3-e7a9f97ffba7" path="/var/lib/kubelet/pods/4875d982-49ff-4402-86c3-e7a9f97ffba7/volumes"
Oct 29 22:58:13.695512 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:58:13.695475    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:58:17.653058 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.653006    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"]
Oct 29 22:58:17.654190 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.654170    4559 cpu_manager.go:414] "containerMap: removing container" podUID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" containerName="istio-proxy"
Oct 29 22:58:17.654190 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.654190    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" containerName="istio-proxy"
Oct 29 22:58:17.654392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.654213    4559 cpu_manager.go:414] "containerMap: removing container" podUID="1c86b737-9464-4a60-9934-64096680f5f3" containerName="maas-api"
Oct 29 22:58:17.654392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.654221    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c86b737-9464-4a60-9934-64096680f5f3" containerName="maas-api"
Oct 29 22:58:17.654392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.654288    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4875d982-49ff-4402-86c3-e7a9f97ffba7" containerName="payload-processing"
Oct 29 22:58:17.654392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.654296    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4875d982-49ff-4402-86c3-e7a9f97ffba7" containerName="payload-processing"
Oct 29 22:58:17.654392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.654336    4559 cpu_manager.go:414] "containerMap: removing container" podUID="08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3" containerName="payload-pre-processing"
Oct 29 22:58:17.654392 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.654344    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3" containerName="payload-pre-processing"
Oct 29 22:58:17.654699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.654517    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" containerName="istio-proxy"
Oct 29 22:58:17.654699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.654531    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c86b737-9464-4a60-9934-64096680f5f3" containerName="maas-api"
Oct 29 22:58:17.654699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.654552    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" containerName="istio-proxy"
Oct 29 22:58:17.654699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.654567    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4875d982-49ff-4402-86c3-e7a9f97ffba7" containerName="payload-processing"
Oct 29 22:58:17.654699 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.654584    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f8dc00-f9d2-4a1c-b70e-b8b2e2c646c3" containerName="payload-pre-processing"
Oct 29 22:58:17.661764 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.661736    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"]
Oct 29 22:58:17.661956 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.661779    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"]
Oct 29 22:58:17.662730 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.662707    4559 cpu_manager.go:414] "containerMap: removing container" podUID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" containerName="istio-proxy"
Oct 29 22:58:17.662730 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.662729    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616f48d5-a65e-48ae-a0e4-b2d3f61d72d2" containerName="istio-proxy"
Oct 29 22:58:17.663336 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.663317    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"
Oct 29 22:58:17.667552 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.667530    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"]
Oct 29 22:58:17.667671 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.667636    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"
Oct 29 22:58:17.684141 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.684110    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4796a9\""
Oct 29 22:58:17.684415 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.684258    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4796a9-dockercfg-bh578\""
Oct 29 22:58:17.742498 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.741878    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7479aed-ec13-4fcc-b119-8a07a523608b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8\" (UID: \"a7479aed-ec13-4fcc-b119-8a07a523608b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"
Oct 29 22:58:17.742498 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.742210    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j49pl\" (UniqueName: \"kubernetes.io/projected/a7479aed-ec13-4fcc-b119-8a07a523608b-kube-api-access-j49pl\") pod \"payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8\" (UID: \"a7479aed-ec13-4fcc-b119-8a07a523608b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"
Oct 29 22:58:17.847071 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.845040    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj\" (UID: \"d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"
Oct 29 22:58:17.847071 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.845194    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lpm6\" (UniqueName: \"kubernetes.io/projected/d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8-kube-api-access-4lpm6\") pod \"payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj\" (UID: \"d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"
Oct 29 22:58:17.847071 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.845296    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j49pl\" (UniqueName: \"kubernetes.io/projected/a7479aed-ec13-4fcc-b119-8a07a523608b-kube-api-access-j49pl\") pod \"payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8\" (UID: \"a7479aed-ec13-4fcc-b119-8a07a523608b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"
Oct 29 22:58:17.847071 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.845467    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7479aed-ec13-4fcc-b119-8a07a523608b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8\" (UID: \"a7479aed-ec13-4fcc-b119-8a07a523608b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"
Oct 29 22:58:17.847071 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.846681    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7479aed-ec13-4fcc-b119-8a07a523608b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8\" (UID: \"a7479aed-ec13-4fcc-b119-8a07a523608b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"
Oct 29 22:58:17.913227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.910954    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j49pl\" (UniqueName: \"kubernetes.io/projected/a7479aed-ec13-4fcc-b119-8a07a523608b-kube-api-access-j49pl\") pod \"payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8\" (UID: \"a7479aed-ec13-4fcc-b119-8a07a523608b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"
Oct 29 22:58:17.947226 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.947177    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj\" (UID: \"d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"
Oct 29 22:58:17.947648 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.947618    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lpm6\" (UniqueName: \"kubernetes.io/projected/d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8-kube-api-access-4lpm6\") pod \"payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj\" (UID: \"d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"
Oct 29 22:58:17.949362 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.949335    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj\" (UID: \"d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"
Oct 29 22:58:17.969189 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:17.969150    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lpm6\" (UniqueName: \"kubernetes.io/projected/d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8-kube-api-access-4lpm6\") pod \"payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj\" (UID: \"d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"
Oct 29 22:58:18.002115 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:18.002069    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"
Oct 29 22:58:18.012714 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:18.012677    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"
Oct 29 22:58:18.377777 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:58:18.377739    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7479aed_ec13_4fcc_b119_8a07a523608b.slice/crio-a63f9b4e68bda36ce38a93eb656b81ca6765a0a1a8200f5c55de3b1d273b7115 WatchSource:0}: Error finding container a63f9b4e68bda36ce38a93eb656b81ca6765a0a1a8200f5c55de3b1d273b7115: Status 404 returned error can't find the container with id a63f9b4e68bda36ce38a93eb656b81ca6765a0a1a8200f5c55de3b1d273b7115
Oct 29 22:58:18.379786 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:18.379720    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"]
Oct 29 22:58:18.393331 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:18.393309    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"]
Oct 29 22:58:18.401618 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:58:18.401580    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89cfeb6_c1a8_478a_859a_f3d2dbcbfbf8.slice/crio-11bcb75a194eb5f02111078e20d35e4d72cf22bcba24fb1c96583eb271b35bef WatchSource:0}: Error finding container 11bcb75a194eb5f02111078e20d35e4d72cf22bcba24fb1c96583eb271b35bef: Status 404 returned error can't find the container with id 11bcb75a194eb5f02111078e20d35e4d72cf22bcba24fb1c96583eb271b35bef
Oct 29 22:58:18.723455 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:18.712374    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8" event={"ID":"a7479aed-ec13-4fcc-b119-8a07a523608b","Type":"ContainerStarted","Data":"79dabdefdfd5ef04d978485579ba15e7ea9ac4bae8e52f22c214d48a056bdfc3"}
Oct 29 22:58:18.723455 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:18.712452    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8" event={"ID":"a7479aed-ec13-4fcc-b119-8a07a523608b","Type":"ContainerStarted","Data":"a63f9b4e68bda36ce38a93eb656b81ca6765a0a1a8200f5c55de3b1d273b7115"}
Oct 29 22:58:18.723455 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:18.714085    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"
Oct 29 22:58:18.737800 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:18.730404    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj" event={"ID":"d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8","Type":"ContainerStarted","Data":"92c6f0a056b33ba02c27a2f4a53abc2ab506fe6d1cad9b46e5667a3bc1ccaf27"}
Oct 29 22:58:18.737800 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:18.730472    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj" event={"ID":"d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8","Type":"ContainerStarted","Data":"11bcb75a194eb5f02111078e20d35e4d72cf22bcba24fb1c96583eb271b35bef"}
Oct 29 22:58:18.737800 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:18.730520    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"
Oct 29 22:58:18.777430 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:18.774496    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8" podStartSLOduration=1.774470681 podStartE2EDuration="1.774470681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:18.753221712 +0000 UTC m=+2637.664665266" watchObservedRunningTime="2026-10-29 22:58:18.774470681 +0000 UTC m=+2637.685914235"
Oct 29 22:58:18.806439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:18.806387    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj" podStartSLOduration=1.8063716410000001 podStartE2EDuration="1.806371641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:18.802649963 +0000 UTC m=+2637.714093513" watchObservedRunningTime="2026-10-29 22:58:18.806371641 +0000 UTC m=+2637.717815195"
Oct 29 22:58:19.501832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:19.501232    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"]
Oct 29 22:58:19.501832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:19.501532    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm" podUID="08448bc0-6fde-4460-bdf7-d65409faa732" containerName="istio-proxy" containerID="cri-o://9b3cac06c5349688137a7896340b11dc9cd33a15536c8684df48bcf6422e7395" gracePeriod=30
Oct 29 22:58:25.004383 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.004350    4559 generic.go:356] "Generic (PLEG): container finished" podID="08448bc0-6fde-4460-bdf7-d65409faa732" containerID="9b3cac06c5349688137a7896340b11dc9cd33a15536c8684df48bcf6422e7395" exitCode=0
Oct 29 22:58:25.004936 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.004420    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm" event={"ID":"08448bc0-6fde-4460-bdf7-d65409faa732","Type":"ContainerDied","Data":"9b3cac06c5349688137a7896340b11dc9cd33a15536c8684df48bcf6422e7395"}
Oct 29 22:58:25.004936 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.004517    4559 scope.go:122] "RemoveContainer" containerID="8096bfd682074ff9db2696aacd1b98126b100ed8cc05bf544a005060fb106b97"
Oct 29 22:58:25.202459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.202430    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:58:25.227456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.227427    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-workload-certs\") pod \"08448bc0-6fde-4460-bdf7-d65409faa732\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") "
Oct 29 22:58:25.227663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.227473    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08448bc0-6fde-4460-bdf7-d65409faa732-istio-token\" (UniqueName: \"kubernetes.io/projected/08448bc0-6fde-4460-bdf7-d65409faa732-istio-token\") pod \"08448bc0-6fde-4460-bdf7-d65409faa732\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") "
Oct 29 22:58:25.227663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.227504    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08448bc0-6fde-4460-bdf7-d65409faa732-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08448bc0-6fde-4460-bdf7-d65409faa732-istiod-ca-cert\") pod \"08448bc0-6fde-4460-bdf7-d65409faa732\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") "
Oct 29 22:58:25.227663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.227580    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-workload-socket\") pod \"08448bc0-6fde-4460-bdf7-d65409faa732\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") "
Oct 29 22:58:25.227663 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.227655    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08448bc0-6fde-4460-bdf7-d65409faa732-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08448bc0-6fde-4460-bdf7-d65409faa732-istio-podinfo\") pod \"08448bc0-6fde-4460-bdf7-d65409faa732\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") "
Oct 29 22:58:25.227975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.227747    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-workload-certs" pod "08448bc0-6fde-4460-bdf7-d65409faa732" (UID: "08448bc0-6fde-4460-bdf7-d65409faa732"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:25.227975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.227764    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-credential-socket\") pod \"08448bc0-6fde-4460-bdf7-d65409faa732\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") "
Oct 29 22:58:25.227975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.227881    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-istio-envoy\") pod \"08448bc0-6fde-4460-bdf7-d65409faa732\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") "
Oct 29 22:58:25.227975 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.227910    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08448bc0-6fde-4460-bdf7-d65409faa732-kube-api-access-xbplh\" (UniqueName: \"kubernetes.io/projected/08448bc0-6fde-4460-bdf7-d65409faa732-kube-api-access-xbplh\") pod \"08448bc0-6fde-4460-bdf7-d65409faa732\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") "
Oct 29 22:58:25.228208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.227985    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-workload-socket" pod "08448bc0-6fde-4460-bdf7-d65409faa732" (UID: "08448bc0-6fde-4460-bdf7-d65409faa732"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:25.228208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.228048    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-istio-data\") pod \"08448bc0-6fde-4460-bdf7-d65409faa732\" (UID: \"08448bc0-6fde-4460-bdf7-d65409faa732\") "
Oct 29 22:58:25.228208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.228163    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-credential-socket" pod "08448bc0-6fde-4460-bdf7-d65409faa732" (UID: "08448bc0-6fde-4460-bdf7-d65409faa732"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:25.228208 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.228083    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08448bc0-6fde-4460-bdf7-d65409faa732-istiod-ca-cert" pod "08448bc0-6fde-4460-bdf7-d65409faa732" (UID: "08448bc0-6fde-4460-bdf7-d65409faa732"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:25.228707 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.228439    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-istio-data" pod "08448bc0-6fde-4460-bdf7-d65409faa732" (UID: "08448bc0-6fde-4460-bdf7-d65409faa732"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:25.231390 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.229306    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08448bc0-6fde-4460-bdf7-d65409faa732-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:25.231390 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.229332    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:25.231390 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.229349    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:25.231390 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.229362    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:25.231390 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.229375    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:25.231772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.231647    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-istio-envoy" pod "08448bc0-6fde-4460-bdf7-d65409faa732" (UID: "08448bc0-6fde-4460-bdf7-d65409faa732"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:25.231772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.231665    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08448bc0-6fde-4460-bdf7-d65409faa732-istio-token" pod "08448bc0-6fde-4460-bdf7-d65409faa732" (UID: "08448bc0-6fde-4460-bdf7-d65409faa732"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:25.231772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.231712    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08448bc0-6fde-4460-bdf7-d65409faa732-kube-api-access-xbplh" pod "08448bc0-6fde-4460-bdf7-d65409faa732" (UID: "08448bc0-6fde-4460-bdf7-d65409faa732"). InnerVolumeSpecName "kube-api-access-xbplh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:25.233008 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.232985    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08448bc0-6fde-4460-bdf7-d65409faa732-istio-podinfo" pod "08448bc0-6fde-4460-bdf7-d65409faa732" (UID: "08448bc0-6fde-4460-bdf7-d65409faa732"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:58:25.409891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.398195    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08448bc0-6fde-4460-bdf7-d65409faa732-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:25.409891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.398237    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08448bc0-6fde-4460-bdf7-d65409faa732-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:25.409891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.398251    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbplh\" (UniqueName: \"kubernetes.io/projected/08448bc0-6fde-4460-bdf7-d65409faa732-kube-api-access-xbplh\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:25.409891 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:25.398265    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08448bc0-6fde-4460-bdf7-d65409faa732-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:26.015550 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:26.015506    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"
Oct 29 22:58:26.016059 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:26.015553    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm" event={"ID":"08448bc0-6fde-4460-bdf7-d65409faa732","Type":"ContainerDied","Data":"057c8bbaf4a3f021e50d5f523fa9a3c2ba70932c186c6f49913f9d8fee139a36"}
Oct 29 22:58:26.016059 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:26.015622    4559 scope.go:122] "RemoveContainer" containerID="9b3cac06c5349688137a7896340b11dc9cd33a15536c8684df48bcf6422e7395"
Oct 29 22:58:26.057096 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:26.057045    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:26.057293 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:26.057118    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm" podUID="08448bc0-6fde-4460-bdf7-d65409faa732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:26.078014 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:26.077978    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"]
Oct 29 22:58:26.087830 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:26.087786    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a962d7-openshift-default-6dbc7cd94c-625mm"]
Oct 29 22:58:27.230350 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:27.230316    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08448bc0-6fde-4460-bdf7-d65409faa732" path="/var/lib/kubelet/pods/08448bc0-6fde-4460-bdf7-d65409faa732/volumes"
Oct 29 22:58:30.005886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:30.005785    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"
Oct 29 22:58:30.119326 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:30.119289    4559 generic.go:356] "Generic (PLEG): container finished" podID="ea391d71-881c-41fb-bb0d-ee9d03a681d0" containerID="7b01695e3676eae49a793a3963e36449026fef50438f5c10a7271ed37e14c966" exitCode=0
Oct 29 22:58:30.119529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:30.119362    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv" event={"ID":"ea391d71-881c-41fb-bb0d-ee9d03a681d0","Type":"ContainerDied","Data":"7b01695e3676eae49a793a3963e36449026fef50438f5c10a7271ed37e14c966"}
Oct 29 22:58:30.119948 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:30.119930    4559 scope.go:122] "RemoveContainer" containerID="7b01695e3676eae49a793a3963e36449026fef50438f5c10a7271ed37e14c966"
Oct 29 22:58:30.122061 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:30.122023    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:58:30.122160 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:30.122116    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:58:30.122220 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:30.122172    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:58:30.457998 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:30.457962    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"]
Oct 29 22:58:30.928428 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:30.928390    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"
Oct 29 22:58:31.130873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:31.130830    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv" event={"ID":"ea391d71-881c-41fb-bb0d-ee9d03a681d0","Type":"ContainerStarted","Data":"2fa78c948318a3cb204f1871757525be9943864c59acee97bca4f15f686c7f23"}
Oct 29 22:58:31.260352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:31.260253    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 29 22:58:31.260638 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:31.260605    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv" podUID="ea391d71-881c-41fb-bb0d-ee9d03a681d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 29 22:58:31.327163 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:31.327119    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:58:31.570010 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:31.569213    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"]
Oct 29 22:58:31.576404 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:31.576365    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"]
Oct 29 22:58:32.283850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:32.257474    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 29 22:58:32.283850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:32.257536    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv" podUID="ea391d71-881c-41fb-bb0d-ee9d03a681d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 29 22:58:33.257423 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:33.257290    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 29 22:58:33.257423 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:33.257361    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv" podUID="ea391d71-881c-41fb-bb0d-ee9d03a681d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 29 22:58:34.262846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:34.262533    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 29 22:58:34.262846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:34.262610    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv" podUID="ea391d71-881c-41fb-bb0d-ee9d03a681d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 29 22:58:35.259211 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:35.259170    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 29 22:58:35.259541 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:35.259497    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv" podUID="ea391d71-881c-41fb-bb0d-ee9d03a681d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 29 22:58:35.461115 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:35.460875    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw" podUID="98ccb29b-516f-4caf-b86f-df0d49744ac1" containerName="payload-processing" containerID="cri-o://3accc652d0c7f629cfe477d0f26ad3cb8ffde01d39e703a449633d338bced3a7" gracePeriod=25
Oct 29 22:58:35.871563 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:35.871471    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"
Oct 29 22:58:35.896601 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:35.896569    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ccb29b-516f-4caf-b86f-df0d49744ac1-kube-api-access-j4lql\" (UniqueName: \"kubernetes.io/projected/98ccb29b-516f-4caf-b86f-df0d49744ac1-kube-api-access-j4lql\") pod \"98ccb29b-516f-4caf-b86f-df0d49744ac1\" (UID: \"98ccb29b-516f-4caf-b86f-df0d49744ac1\") "
Oct 29 22:58:35.897021 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:35.896990    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ccb29b-516f-4caf-b86f-df0d49744ac1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ccb29b-516f-4caf-b86f-df0d49744ac1-plugins-config-volume\") pod \"98ccb29b-516f-4caf-b86f-df0d49744ac1\" (UID: \"98ccb29b-516f-4caf-b86f-df0d49744ac1\") "
Oct 29 22:58:35.897529 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:35.897505    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ccb29b-516f-4caf-b86f-df0d49744ac1-plugins-config-volume" pod "98ccb29b-516f-4caf-b86f-df0d49744ac1" (UID: "98ccb29b-516f-4caf-b86f-df0d49744ac1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:35.897934 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:35.897915    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ccb29b-516f-4caf-b86f-df0d49744ac1-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:35.899728 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:35.899700    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ccb29b-516f-4caf-b86f-df0d49744ac1-kube-api-access-j4lql" pod "98ccb29b-516f-4caf-b86f-df0d49744ac1" (UID: "98ccb29b-516f-4caf-b86f-df0d49744ac1"). InnerVolumeSpecName "kube-api-access-j4lql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:36.051840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:36.047013    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4lql\" (UniqueName: \"kubernetes.io/projected/98ccb29b-516f-4caf-b86f-df0d49744ac1-kube-api-access-j4lql\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:36.214680 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:36.214644    4559 generic.go:356] "Generic (PLEG): container finished" podID="98ccb29b-516f-4caf-b86f-df0d49744ac1" containerID="3accc652d0c7f629cfe477d0f26ad3cb8ffde01d39e703a449633d338bced3a7" exitCode=0
Oct 29 22:58:36.214914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:36.214699    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"
Oct 29 22:58:36.214914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:36.214725    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw" event={"ID":"98ccb29b-516f-4caf-b86f-df0d49744ac1","Type":"ContainerDied","Data":"3accc652d0c7f629cfe477d0f26ad3cb8ffde01d39e703a449633d338bced3a7"}
Oct 29 22:58:36.214914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:36.214779    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw" event={"ID":"98ccb29b-516f-4caf-b86f-df0d49744ac1","Type":"ContainerDied","Data":"9c3f1fdd7122ce9f38c9fd2d8f195d6cb60d56cd64ee38c8b0e9ebd10c6c733b"}
Oct 29 22:58:36.214914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:36.214791    4559 scope.go:122] "RemoveContainer" containerID="3accc652d0c7f629cfe477d0f26ad3cb8ffde01d39e703a449633d338bced3a7"
Oct 29 22:58:36.226742 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:36.226721    4559 scope.go:122] "RemoveContainer" containerID="3accc652d0c7f629cfe477d0f26ad3cb8ffde01d39e703a449633d338bced3a7"
Oct 29 22:58:36.227053 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:58:36.227033    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3accc652d0c7f629cfe477d0f26ad3cb8ffde01d39e703a449633d338bced3a7\": container with ID starting with 3accc652d0c7f629cfe477d0f26ad3cb8ffde01d39e703a449633d338bced3a7 not found: ID does not exist" containerID="3accc652d0c7f629cfe477d0f26ad3cb8ffde01d39e703a449633d338bced3a7"
Oct 29 22:58:36.227135 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:36.227068    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3accc652d0c7f629cfe477d0f26ad3cb8ffde01d39e703a449633d338bced3a7"} err="failed to get container status \"3accc652d0c7f629cfe477d0f26ad3cb8ffde01d39e703a449633d338bced3a7\": rpc error: code = NotFound desc = could not find container \"3accc652d0c7f629cfe477d0f26ad3cb8ffde01d39e703a449633d338bced3a7\": container with ID starting with 3accc652d0c7f629cfe477d0f26ad3cb8ffde01d39e703a449633d338bced3a7 not found: ID does not exist"
Oct 29 22:58:36.260129 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:36.259612    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 29 22:58:36.260129 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:36.259678    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv" podUID="ea391d71-881c-41fb-bb0d-ee9d03a681d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 29 22:58:36.275993 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:36.275953    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"]
Oct 29 22:58:36.283203 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:36.283176    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-77b9597456-2htkw"]
Oct 29 22:58:36.570139 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:36.570026    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q" podUID="cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc" containerName="payload-processing" containerID="cri-o://a7a1c71a1f5186ca4c4ee25354d88448587bd7a460dfcc26aa6524a8d1e2e857" gracePeriod=25
Oct 29 22:58:36.577924 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:36.577856    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j" podUID="4010ffa9-1ddd-4f51-afc9-eb5ea50bf969" containerName="payload-pre-processing" containerID="cri-o://8417e84617af0c3654b8c1201b371133789fb1ca6dc17adfec48b18351609a1d" gracePeriod=25
Oct 29 22:58:36.975836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:36.975070    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"
Oct 29 22:58:37.086113 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.086077    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc-kube-api-access-qp7xq\" (UniqueName: \"kubernetes.io/projected/cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc-kube-api-access-qp7xq\") pod \"cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc\" (UID: \"cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc\") "
Oct 29 22:58:37.086356 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.086344    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc-plugins-config-volume\") pod \"cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc\" (UID: \"cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc\") "
Oct 29 22:58:37.088134 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.088103    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc-plugins-config-volume" pod "cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc" (UID: "cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:37.092450 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.092418    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc-kube-api-access-qp7xq" pod "cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc" (UID: "cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc"). InnerVolumeSpecName "kube-api-access-qp7xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:37.093763 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.093734    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"
Oct 29 22:58:37.158898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.145947    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:58:37.187903 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.187866    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4010ffa9-1ddd-4f51-afc9-eb5ea50bf969-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4010ffa9-1ddd-4f51-afc9-eb5ea50bf969-plugins-config-volume\") pod \"4010ffa9-1ddd-4f51-afc9-eb5ea50bf969\" (UID: \"4010ffa9-1ddd-4f51-afc9-eb5ea50bf969\") "
Oct 29 22:58:37.188113 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.188055    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4010ffa9-1ddd-4f51-afc9-eb5ea50bf969-kube-api-access-hr5ph\" (UniqueName: \"kubernetes.io/projected/4010ffa9-1ddd-4f51-afc9-eb5ea50bf969-kube-api-access-hr5ph\") pod \"4010ffa9-1ddd-4f51-afc9-eb5ea50bf969\" (UID: \"4010ffa9-1ddd-4f51-afc9-eb5ea50bf969\") "
Oct 29 22:58:37.188797 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.188770    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:37.188797 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.188800    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp7xq\" (UniqueName: \"kubernetes.io/projected/cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc-kube-api-access-qp7xq\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:37.189364 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.189332    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4010ffa9-1ddd-4f51-afc9-eb5ea50bf969-plugins-config-volume" pod "4010ffa9-1ddd-4f51-afc9-eb5ea50bf969" (UID: "4010ffa9-1ddd-4f51-afc9-eb5ea50bf969"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:37.191750 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.191723    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4010ffa9-1ddd-4f51-afc9-eb5ea50bf969-kube-api-access-hr5ph" pod "4010ffa9-1ddd-4f51-afc9-eb5ea50bf969" (UID: "4010ffa9-1ddd-4f51-afc9-eb5ea50bf969"). InnerVolumeSpecName "kube-api-access-hr5ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:37.229005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.228853    4559 generic.go:356] "Generic (PLEG): container finished" podID="cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc" containerID="a7a1c71a1f5186ca4c4ee25354d88448587bd7a460dfcc26aa6524a8d1e2e857" exitCode=0
Oct 29 22:58:37.229005 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.228907    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"
Oct 29 22:58:37.230506 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.230479    4559 generic.go:356] "Generic (PLEG): container finished" podID="4010ffa9-1ddd-4f51-afc9-eb5ea50bf969" containerID="8417e84617af0c3654b8c1201b371133789fb1ca6dc17adfec48b18351609a1d" exitCode=0
Oct 29 22:58:37.230662 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.230587    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"
Oct 29 22:58:37.246575 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.246510    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ccb29b-516f-4caf-b86f-df0d49744ac1" path="/var/lib/kubelet/pods/98ccb29b-516f-4caf-b86f-df0d49744ac1/volumes"
Oct 29 22:58:37.248061 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.248029    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q" event={"ID":"cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc","Type":"ContainerDied","Data":"a7a1c71a1f5186ca4c4ee25354d88448587bd7a460dfcc26aa6524a8d1e2e857"}
Oct 29 22:58:37.248216 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.248096    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q" event={"ID":"cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc","Type":"ContainerDied","Data":"49decfb3344c33d915ec5de439b0d9d4cb0429481ff896ebe0a4b820199bddb9"}
Oct 29 22:58:37.248216 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.248124    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j" event={"ID":"4010ffa9-1ddd-4f51-afc9-eb5ea50bf969","Type":"ContainerDied","Data":"8417e84617af0c3654b8c1201b371133789fb1ca6dc17adfec48b18351609a1d"}
Oct 29 22:58:37.248216 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.248149    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j" event={"ID":"4010ffa9-1ddd-4f51-afc9-eb5ea50bf969","Type":"ContainerDied","Data":"04588957bd1343c959622841b544c9bd9597dbebd7d54f3011aff4438620dbda"}
Oct 29 22:58:37.248216 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.248177    4559 scope.go:122] "RemoveContainer" containerID="a7a1c71a1f5186ca4c4ee25354d88448587bd7a460dfcc26aa6524a8d1e2e857"
Oct 29 22:58:37.259189 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.259157    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 29 22:58:37.259312 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.259253    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv" podUID="ea391d71-881c-41fb-bb0d-ee9d03a681d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 29 22:58:37.275179 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.275152    4559 scope.go:122] "RemoveContainer" containerID="a7a1c71a1f5186ca4c4ee25354d88448587bd7a460dfcc26aa6524a8d1e2e857"
Oct 29 22:58:37.275587 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:58:37.275561    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a1c71a1f5186ca4c4ee25354d88448587bd7a460dfcc26aa6524a8d1e2e857\": container with ID starting with a7a1c71a1f5186ca4c4ee25354d88448587bd7a460dfcc26aa6524a8d1e2e857 not found: ID does not exist" containerID="a7a1c71a1f5186ca4c4ee25354d88448587bd7a460dfcc26aa6524a8d1e2e857"
Oct 29 22:58:37.275664 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.275599    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a1c71a1f5186ca4c4ee25354d88448587bd7a460dfcc26aa6524a8d1e2e857"} err="failed to get container status \"a7a1c71a1f5186ca4c4ee25354d88448587bd7a460dfcc26aa6524a8d1e2e857\": rpc error: code = NotFound desc = could not find container \"a7a1c71a1f5186ca4c4ee25354d88448587bd7a460dfcc26aa6524a8d1e2e857\": container with ID starting with a7a1c71a1f5186ca4c4ee25354d88448587bd7a460dfcc26aa6524a8d1e2e857 not found: ID does not exist"
Oct 29 22:58:37.275664 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.275618    4559 scope.go:122] "RemoveContainer" containerID="8417e84617af0c3654b8c1201b371133789fb1ca6dc17adfec48b18351609a1d"
Oct 29 22:58:37.288053 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.288032    4559 scope.go:122] "RemoveContainer" containerID="8417e84617af0c3654b8c1201b371133789fb1ca6dc17adfec48b18351609a1d"
Oct 29 22:58:37.288358 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:58:37.288334    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8417e84617af0c3654b8c1201b371133789fb1ca6dc17adfec48b18351609a1d\": container with ID starting with 8417e84617af0c3654b8c1201b371133789fb1ca6dc17adfec48b18351609a1d not found: ID does not exist" containerID="8417e84617af0c3654b8c1201b371133789fb1ca6dc17adfec48b18351609a1d"
Oct 29 22:58:37.288439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.288366    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8417e84617af0c3654b8c1201b371133789fb1ca6dc17adfec48b18351609a1d"} err="failed to get container status \"8417e84617af0c3654b8c1201b371133789fb1ca6dc17adfec48b18351609a1d\": rpc error: code = NotFound desc = could not find container \"8417e84617af0c3654b8c1201b371133789fb1ca6dc17adfec48b18351609a1d\": container with ID starting with 8417e84617af0c3654b8c1201b371133789fb1ca6dc17adfec48b18351609a1d not found: ID does not exist"
Oct 29 22:58:37.290750 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.290722    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4010ffa9-1ddd-4f51-afc9-eb5ea50bf969-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:37.290750 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.290750    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr5ph\" (UniqueName: \"kubernetes.io/projected/4010ffa9-1ddd-4f51-afc9-eb5ea50bf969-kube-api-access-hr5ph\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:58:37.303299 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.303268    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"]
Oct 29 22:58:37.310687 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.310659    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-06a95f-6c459f96db-6pb7q"]
Oct 29 22:58:37.343747 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.343703    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"]
Oct 29 22:58:37.347865 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:37.347840    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-06a95f-7c5bb67c55-dkr8j"]
Oct 29 22:58:38.263707 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:38.263674    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:58:38.265022 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:38.265000    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:58:39.322886 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:39.322793    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4010ffa9-1ddd-4f51-afc9-eb5ea50bf969" path="/var/lib/kubelet/pods/4010ffa9-1ddd-4f51-afc9-eb5ea50bf969/volumes"
Oct 29 22:58:39.324846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:39.323681    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc" path="/var/lib/kubelet/pods/cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc/volumes"
Oct 29 22:58:41.431842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:41.430614    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 22:58:43.767470 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:58:43.767430    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:58:47.740836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.727136    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"]
Oct 29 22:58:47.740836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.728380    4559 cpu_manager.go:414] "containerMap: removing container" podUID="cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc" containerName="payload-processing"
Oct 29 22:58:47.740836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.728400    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc" containerName="payload-processing"
Oct 29 22:58:47.740836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.728439    4559 cpu_manager.go:414] "containerMap: removing container" podUID="08448bc0-6fde-4460-bdf7-d65409faa732" containerName="istio-proxy"
Oct 29 22:58:47.740836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.728447    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08448bc0-6fde-4460-bdf7-d65409faa732" containerName="istio-proxy"
Oct 29 22:58:47.740836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.728462    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4010ffa9-1ddd-4f51-afc9-eb5ea50bf969" containerName="payload-pre-processing"
Oct 29 22:58:47.740836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.728470    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4010ffa9-1ddd-4f51-afc9-eb5ea50bf969" containerName="payload-pre-processing"
Oct 29 22:58:47.740836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.728507    4559 cpu_manager.go:414] "containerMap: removing container" podUID="08448bc0-6fde-4460-bdf7-d65409faa732" containerName="istio-proxy"
Oct 29 22:58:47.740836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.728515    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08448bc0-6fde-4460-bdf7-d65409faa732" containerName="istio-proxy"
Oct 29 22:58:47.740836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.728549    4559 cpu_manager.go:414] "containerMap: removing container" podUID="98ccb29b-516f-4caf-b86f-df0d49744ac1" containerName="payload-processing"
Oct 29 22:58:47.740836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.728556    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ccb29b-516f-4caf-b86f-df0d49744ac1" containerName="payload-processing"
Oct 29 22:58:47.740836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.728709    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ccb29b-516f-4caf-b86f-df0d49744ac1" containerName="payload-processing"
Oct 29 22:58:47.740836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.728724    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc9efeeb-b2b3-48a6-8e7f-e10b493a19fc" containerName="payload-processing"
Oct 29 22:58:47.740836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.728742    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="08448bc0-6fde-4460-bdf7-d65409faa732" containerName="istio-proxy"
Oct 29 22:58:47.740836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.728752    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4010ffa9-1ddd-4f51-afc9-eb5ea50bf969" containerName="payload-pre-processing"
Oct 29 22:58:47.740836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.728767    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="08448bc0-6fde-4460-bdf7-d65409faa732" containerName="istio-proxy"
Oct 29 22:58:47.740836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.735181    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:47.780900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.780866    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-80248b-openshift-default-dockercfg-2ndrz\""
Oct 29 22:58:47.781174 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.781147    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"]
Oct 29 22:58:47.851836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.841646    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-token\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:47.851836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.841728    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgw2f\" (UniqueName: \"kubernetes.io/projected/6f1b07f5-c3bd-4382-9318-2234d4e9093e-kube-api-access-dgw2f\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:47.991907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.990986    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-data\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:47.991907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.991098    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-workload-certs\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:47.991907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.991147    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-credential-socket\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:47.991907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.991251    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-envoy\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:47.991907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.991390    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-podinfo\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:47.991907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.991450    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-workload-socket\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:47.991907 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:47.991496    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istiod-ca-cert\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.105837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.098082    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-podinfo\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.105837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.098164    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-workload-socket\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.105837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.098211    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istiod-ca-cert\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.105837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.098291    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-token\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.105837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.098331    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgw2f\" (UniqueName: \"kubernetes.io/projected/6f1b07f5-c3bd-4382-9318-2234d4e9093e-kube-api-access-dgw2f\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.105837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.098459    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-data\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.105837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.098525    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-workload-certs\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.105837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.098563    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-credential-socket\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.105837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.098647    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-envoy\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.105837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.100191    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-workload-socket\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.105837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.100624    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istiod-ca-cert\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.105837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.103272    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-workload-certs\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.105837 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.103522    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-credential-socket\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.194221 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.182006    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-data\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.224933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.224851    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgw2f\" (UniqueName: \"kubernetes.io/projected/6f1b07f5-c3bd-4382-9318-2234d4e9093e-kube-api-access-dgw2f\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.388738 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.380580    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-envoy\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.431801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.430513    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-podinfo\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.431801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.430949    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-token\") pod \"e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") " pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.432751 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.432518    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:48.872791 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:58:48.872748    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1b07f5_c3bd_4382_9318_2234d4e9093e.slice/crio-125ee54355607d87e323cd5eafa0268e345c876f04db1ceb6457123d927736b7 WatchSource:0}: Error finding container 125ee54355607d87e323cd5eafa0268e345c876f04db1ceb6457123d927736b7: Status 404 returned error can't find the container with id 125ee54355607d87e323cd5eafa0268e345c876f04db1ceb6457123d927736b7
Oct 29 22:58:48.873780 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.873729    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"]
Oct 29 22:58:48.876520 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.876479    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:58:48.876601 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.876572    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:58:48.876641 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.876616    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:58:48.958189 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:48.949597    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs" event={"ID":"6f1b07f5-c3bd-4382-9318-2234d4e9093e","Type":"ContainerStarted","Data":"125ee54355607d87e323cd5eafa0268e345c876f04db1ceb6457123d927736b7"}
Oct 29 22:58:49.979175 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:49.979136    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs" event={"ID":"6f1b07f5-c3bd-4382-9318-2234d4e9093e","Type":"ContainerStarted","Data":"be840790c395da14e994e5eed7278ff4ab0f9d97d887505937f7ee3cc9f1e3ff"}
Oct 29 22:58:50.050282 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:50.050222    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs" podStartSLOduration=3.050204709 podStartE2EDuration="3.050204709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:50.036373702 +0000 UTC m=+2668.947817259" watchObservedRunningTime="2026-10-29 22:58:50.050204709 +0000 UTC m=+2668.961648262"
Oct 29 22:58:50.483113 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:50.483072    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:50.489869 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:50.489825    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:50.989352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:50.989316    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:50.990559 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:50.990534    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 22:58:53.804878 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:53.797883    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"]
Oct 29 22:58:53.807962 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:53.807934    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"]
Oct 29 22:58:53.808229 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:53.808217    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"
Oct 29 22:58:53.830286 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:53.830247    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-85862e\""
Oct 29 22:58:53.835747 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:53.835712    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-85862e-dockercfg-8l9jz\""
Oct 29 22:58:53.920771 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:53.920734    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5789ad1-f64c-4eb8-9a38-49da59afb179-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw\" (UID: \"d5789ad1-f64c-4eb8-9a38-49da59afb179\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"
Oct 29 22:58:53.920985 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:53.920834    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcjqq\" (UniqueName: \"kubernetes.io/projected/d5789ad1-f64c-4eb8-9a38-49da59afb179-kube-api-access-rcjqq\") pod \"payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw\" (UID: \"d5789ad1-f64c-4eb8-9a38-49da59afb179\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"
Oct 29 22:58:54.022597 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:54.022563    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5789ad1-f64c-4eb8-9a38-49da59afb179-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw\" (UID: \"d5789ad1-f64c-4eb8-9a38-49da59afb179\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"
Oct 29 22:58:54.022780 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:54.022636    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcjqq\" (UniqueName: \"kubernetes.io/projected/d5789ad1-f64c-4eb8-9a38-49da59afb179-kube-api-access-rcjqq\") pod \"payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw\" (UID: \"d5789ad1-f64c-4eb8-9a38-49da59afb179\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"
Oct 29 22:58:54.023347 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:54.023327    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5789ad1-f64c-4eb8-9a38-49da59afb179-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw\" (UID: \"d5789ad1-f64c-4eb8-9a38-49da59afb179\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"
Oct 29 22:58:54.043201 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:54.043164    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcjqq\" (UniqueName: \"kubernetes.io/projected/d5789ad1-f64c-4eb8-9a38-49da59afb179-kube-api-access-rcjqq\") pod \"payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw\" (UID: \"d5789ad1-f64c-4eb8-9a38-49da59afb179\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"
Oct 29 22:58:54.140710 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:54.140667    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"
Oct 29 22:58:54.481463 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:54.480526    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"]
Oct 29 22:58:54.524265 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:58:54.521460    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5789ad1_f64c_4eb8_9a38_49da59afb179.slice/crio-30de434d54c8524486d85e0d481e2ec21baf2cd2474d2068dfc8aafb20791efc WatchSource:0}: Error finding container 30de434d54c8524486d85e0d481e2ec21baf2cd2474d2068dfc8aafb20791efc: Status 404 returned error can't find the container with id 30de434d54c8524486d85e0d481e2ec21baf2cd2474d2068dfc8aafb20791efc
Oct 29 22:58:55.025791 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.025746    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw" event={"ID":"d5789ad1-f64c-4eb8-9a38-49da59afb179","Type":"ContainerStarted","Data":"1c412d2394b6939b47f17669f359f9b4fbba61791a651d42bb5660c2e6acb45f"}
Oct 29 22:58:55.026380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.025841    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw" event={"ID":"d5789ad1-f64c-4eb8-9a38-49da59afb179","Type":"ContainerStarted","Data":"30de434d54c8524486d85e0d481e2ec21baf2cd2474d2068dfc8aafb20791efc"}
Oct 29 22:58:55.026380 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.025872    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"
Oct 29 22:58:55.090861 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.090781    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw" podStartSLOduration=2.090760574 podStartE2EDuration="2.090760574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:55.089212552 +0000 UTC m=+2674.000656134" watchObservedRunningTime="2026-10-29 22:58:55.090760574 +0000 UTC m=+2674.002204128"
Oct 29 22:58:55.178987 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.178953    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"]
Oct 29 22:58:55.191474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.185211    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"
Oct 29 22:58:55.205051 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.205011    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-80248b-dockercfg-8c7jz\""
Oct 29 22:58:55.205834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.205717    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-80248b\""
Oct 29 22:58:55.216888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.206701    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"]
Oct 29 22:58:55.216888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.213204    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"
Oct 29 22:58:55.216888 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.213518    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"]
Oct 29 22:58:55.255150 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.253921    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"]
Oct 29 22:58:55.255639 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.255590    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc\" (UID: \"c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"
Oct 29 22:58:55.256126 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.256095    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2prd2\" (UniqueName: \"kubernetes.io/projected/c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40-kube-api-access-2prd2\") pod \"payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc\" (UID: \"c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"
Oct 29 22:58:55.398227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.396110    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2prd2\" (UniqueName: \"kubernetes.io/projected/c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40-kube-api-access-2prd2\") pod \"payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc\" (UID: \"c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"
Oct 29 22:58:55.398227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.396206    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/851f7422-65c1-4771-b581-cc578a3ce43f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d\" (UID: \"851f7422-65c1-4771-b581-cc578a3ce43f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"
Oct 29 22:58:55.398227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.396418    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6r9z\" (UniqueName: \"kubernetes.io/projected/851f7422-65c1-4771-b581-cc578a3ce43f-kube-api-access-m6r9z\") pod \"payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d\" (UID: \"851f7422-65c1-4771-b581-cc578a3ce43f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"
Oct 29 22:58:55.398227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.396598    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc\" (UID: \"c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"
Oct 29 22:58:55.398227 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.397832    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc\" (UID: \"c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"
Oct 29 22:58:55.464235 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.459827    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2prd2\" (UniqueName: \"kubernetes.io/projected/c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40-kube-api-access-2prd2\") pod \"payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc\" (UID: \"c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"
Oct 29 22:58:55.523358 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.523056    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"
Oct 29 22:58:55.553223 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.536741    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6r9z\" (UniqueName: \"kubernetes.io/projected/851f7422-65c1-4771-b581-cc578a3ce43f-kube-api-access-m6r9z\") pod \"payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d\" (UID: \"851f7422-65c1-4771-b581-cc578a3ce43f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"
Oct 29 22:58:55.556540 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.555346    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/851f7422-65c1-4771-b581-cc578a3ce43f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d\" (UID: \"851f7422-65c1-4771-b581-cc578a3ce43f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"
Oct 29 22:58:55.556540 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.556484    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/851f7422-65c1-4771-b581-cc578a3ce43f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d\" (UID: \"851f7422-65c1-4771-b581-cc578a3ce43f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"
Oct 29 22:58:55.698710 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.698669    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6r9z\" (UniqueName: \"kubernetes.io/projected/851f7422-65c1-4771-b581-cc578a3ce43f-kube-api-access-m6r9z\") pod \"payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d\" (UID: \"851f7422-65c1-4771-b581-cc578a3ce43f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"
Oct 29 22:58:55.896304 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:55.879754    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"
Oct 29 22:58:56.140037 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:56.140002    4559 generic.go:356] "Generic (PLEG): container finished" podID="2c3db4b3-0ba7-49d3-9349-327aa6022a71" containerID="2213aca4d64dab8d98d60076dc33ffa470aba170197d61a8e08e169e526f13ff" exitCode=0
Oct 29 22:58:56.141043 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:56.140077    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt" event={"ID":"2c3db4b3-0ba7-49d3-9349-327aa6022a71","Type":"ContainerDied","Data":"2213aca4d64dab8d98d60076dc33ffa470aba170197d61a8e08e169e526f13ff"}
Oct 29 22:58:56.143382 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:56.142569    4559 scope.go:122] "RemoveContainer" containerID="2213aca4d64dab8d98d60076dc33ffa470aba170197d61a8e08e169e526f13ff"
Oct 29 22:58:56.146658 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:56.146575    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:58:56.148350 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:56.146951    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:58:56.148350 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:56.147011    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:58:56.413641 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:58:56.413604    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc460b9f9_ba2f_4acf_9d2f_8ea1dea0ce40.slice/crio-a7f907c6e2c60dc7c9449116a2979ac4f262c6a95b142672a26959ef5c10c27b WatchSource:0}: Error finding container a7f907c6e2c60dc7c9449116a2979ac4f262c6a95b142672a26959ef5c10c27b: Status 404 returned error can't find the container with id a7f907c6e2c60dc7c9449116a2979ac4f262c6a95b142672a26959ef5c10c27b
Oct 29 22:58:56.414706 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:56.414650    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"]
Oct 29 22:58:56.459580 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:58:56.459533    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851f7422_65c1_4771_b581_cc578a3ce43f.slice/crio-90991d519e3faa49d5993daf233b8c1e186feba46610785688de743023283999 WatchSource:0}: Error finding container 90991d519e3faa49d5993daf233b8c1e186feba46610785688de743023283999: Status 404 returned error can't find the container with id 90991d519e3faa49d5993daf233b8c1e186feba46610785688de743023283999
Oct 29 22:58:56.461083 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:56.461055    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"]
Oct 29 22:58:57.173787 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:57.173742    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc" event={"ID":"c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40","Type":"ContainerStarted","Data":"b486bdc2091ec8f5d2b6813409aee037344b2cf17c7daa6b13c7f65094ac1d67"}
Oct 29 22:58:57.174338 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:57.173831    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc" event={"ID":"c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40","Type":"ContainerStarted","Data":"a7f907c6e2c60dc7c9449116a2979ac4f262c6a95b142672a26959ef5c10c27b"}
Oct 29 22:58:57.174338 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:57.174188    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"
Oct 29 22:58:57.176339 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:57.176307    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt" event={"ID":"2c3db4b3-0ba7-49d3-9349-327aa6022a71","Type":"ContainerStarted","Data":"71b77cbf4daed37016865e2287e649cb31c1ae31504f884f42a229b6c400505e"}
Oct 29 22:58:57.178517 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:57.178485    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d" event={"ID":"851f7422-65c1-4771-b581-cc578a3ce43f","Type":"ContainerStarted","Data":"5cec1ddf5dea9ce6dcaa1d9b5d7eeb1c8303847c5ff00f743e4abc991119c741"}
Oct 29 22:58:57.178674 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:57.178548    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"
Oct 29 22:58:57.178674 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:57.178571    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d" event={"ID":"851f7422-65c1-4771-b581-cc578a3ce43f","Type":"ContainerStarted","Data":"90991d519e3faa49d5993daf233b8c1e186feba46610785688de743023283999"}
Oct 29 22:58:57.218788 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:57.218730    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc" podStartSLOduration=2.21871564 podStartE2EDuration="2.21871564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:57.215834633 +0000 UTC m=+2676.127278186" watchObservedRunningTime="2026-10-29 22:58:57.21871564 +0000 UTC m=+2676.130159233"
Oct 29 22:58:57.310919 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:57.310863    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d" podStartSLOduration=2.31080526 podStartE2EDuration="2.31080526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:57.307443215 +0000 UTC m=+2676.218886769" watchObservedRunningTime="2026-10-29 22:58:57.31080526 +0000 UTC m=+2676.222248818"
Oct 29 22:58:57.644313 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:57.644280    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:58:57.649832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:57.649466    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Oct 29 22:58:57.649832 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:57.649535    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt" podUID="2c3db4b3-0ba7-49d3-9349-327aa6022a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Oct 29 22:58:58.640214 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:58.640177    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Oct 29 22:58:58.640877 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:58.640844    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt" podUID="2c3db4b3-0ba7-49d3-9349-327aa6022a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Oct 29 22:58:59.639278 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:59.639235    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Oct 29 22:58:59.639577 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:58:59.639545    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt" podUID="2c3db4b3-0ba7-49d3-9349-327aa6022a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Oct 29 22:59:00.641839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:00.640223    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Oct 29 22:59:00.641839 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:00.640286    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt" podUID="2c3db4b3-0ba7-49d3-9349-327aa6022a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Oct 29 22:59:01.654952 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:01.654916    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:59:01.657046 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:01.657015    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Oct 29 22:59:01.657244 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:01.657220    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt" podUID="2c3db4b3-0ba7-49d3-9349-327aa6022a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Oct 29 22:59:02.629446 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:02.629406    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"]
Oct 29 22:59:02.629692 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:02.629666    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl" podUID="a75464c2-c1ca-4991-8634-7c1f417400b5" containerName="maas-api" containerID="cri-o://a40cf86174a72fa8d8a2faf133a965e12984d0ccfb73cc8838052f1fbe086487" gracePeriod=30
Oct 29 22:59:02.636846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:02.634543    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Oct 29 22:59:02.636846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:02.634618    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt" podUID="2c3db4b3-0ba7-49d3-9349-327aa6022a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Oct 29 22:59:02.805917 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:02.805873    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl" podUID="a75464c2-c1ca-4991-8634-7c1f417400b5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.96:8443/health\": dial tcp 10.134.0.96:8443: connect: connection refused"
Oct 29 22:59:03.241563 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:03.225174    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"]
Oct 29 22:59:03.359217 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:03.316325    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj" podUID="d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8" containerName="payload-processing" containerID="cri-o://92c6f0a056b33ba02c27a2f4a53abc2ab506fe6d1cad9b46e5667a3bc1ccaf27" gracePeriod=30
Oct 29 22:59:03.571848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:03.548970    4559 generic.go:356] "Generic (PLEG): container finished" podID="a75464c2-c1ca-4991-8634-7c1f417400b5" containerID="a40cf86174a72fa8d8a2faf133a965e12984d0ccfb73cc8838052f1fbe086487" exitCode=0
Oct 29 22:59:03.592238 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:03.577514    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl" event={"ID":"a75464c2-c1ca-4991-8634-7c1f417400b5","Type":"ContainerDied","Data":"a40cf86174a72fa8d8a2faf133a965e12984d0ccfb73cc8838052f1fbe086487"}
Oct 29 22:59:03.693704 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:03.689192    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:59:03.694787 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:03.694748    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 22:59:03.778997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:03.754521    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"]
Oct 29 22:59:03.778997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:03.754774    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8" podUID="a7479aed-ec13-4fcc-b119-8a07a523608b" containerName="payload-pre-processing" containerID="cri-o://79dabdefdfd5ef04d978485579ba15e7ea9ac4bae8e52f22c214d48a056bdfc3" gracePeriod=30
Oct 29 22:59:03.909840 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:03.896801    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:59:04.032893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.031160    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a75464c2-c1ca-4991-8634-7c1f417400b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a75464c2-c1ca-4991-8634-7c1f417400b5-metrics-certs\") pod \"a75464c2-c1ca-4991-8634-7c1f417400b5\" (UID: \"a75464c2-c1ca-4991-8634-7c1f417400b5\") "
Oct 29 22:59:04.032893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.031220    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75464c2-c1ca-4991-8634-7c1f417400b5-kube-api-access-c87b6\" (UniqueName: \"kubernetes.io/projected/a75464c2-c1ca-4991-8634-7c1f417400b5-kube-api-access-c87b6\") pod \"a75464c2-c1ca-4991-8634-7c1f417400b5\" (UID: \"a75464c2-c1ca-4991-8634-7c1f417400b5\") "
Oct 29 22:59:04.032893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.031309    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a75464c2-c1ca-4991-8634-7c1f417400b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75464c2-c1ca-4991-8634-7c1f417400b5-maas-api-tls\") pod \"a75464c2-c1ca-4991-8634-7c1f417400b5\" (UID: \"a75464c2-c1ca-4991-8634-7c1f417400b5\") "
Oct 29 22:59:04.032893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.031461    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a75464c2-c1ca-4991-8634-7c1f417400b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75464c2-c1ca-4991-8634-7c1f417400b5-openshift-service-ca\") pod \"a75464c2-c1ca-4991-8634-7c1f417400b5\" (UID: \"a75464c2-c1ca-4991-8634-7c1f417400b5\") "
Oct 29 22:59:04.038624 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.033484    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75464c2-c1ca-4991-8634-7c1f417400b5-openshift-service-ca" pod "a75464c2-c1ca-4991-8634-7c1f417400b5" (UID: "a75464c2-c1ca-4991-8634-7c1f417400b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:04.135735 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.135696    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75464c2-c1ca-4991-8634-7c1f417400b5-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.145507 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.145457    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75464c2-c1ca-4991-8634-7c1f417400b5-metrics-certs" pod "a75464c2-c1ca-4991-8634-7c1f417400b5" (UID: "a75464c2-c1ca-4991-8634-7c1f417400b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:04.176284 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.175728    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75464c2-c1ca-4991-8634-7c1f417400b5-kube-api-access-c87b6" pod "a75464c2-c1ca-4991-8634-7c1f417400b5" (UID: "a75464c2-c1ca-4991-8634-7c1f417400b5"). InnerVolumeSpecName "kube-api-access-c87b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:04.180900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.180828    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75464c2-c1ca-4991-8634-7c1f417400b5-maas-api-tls" pod "a75464c2-c1ca-4991-8634-7c1f417400b5" (UID: "a75464c2-c1ca-4991-8634-7c1f417400b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:04.239045 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.238955    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75464c2-c1ca-4991-8634-7c1f417400b5-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.239045 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.238995    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a75464c2-c1ca-4991-8634-7c1f417400b5-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.239045 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.239010    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c87b6\" (UniqueName: \"kubernetes.io/projected/a75464c2-c1ca-4991-8634-7c1f417400b5-kube-api-access-c87b6\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.488234 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.487911    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"
Oct 29 22:59:04.557092 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.557062    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"
Oct 29 22:59:04.566787 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.566743    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7479aed-ec13-4fcc-b119-8a07a523608b-kube-api-access-j49pl\" (UniqueName: \"kubernetes.io/projected/a7479aed-ec13-4fcc-b119-8a07a523608b-kube-api-access-j49pl\") pod \"a7479aed-ec13-4fcc-b119-8a07a523608b\" (UID: \"a7479aed-ec13-4fcc-b119-8a07a523608b\") "
Oct 29 22:59:04.567110 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.567088    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7479aed-ec13-4fcc-b119-8a07a523608b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7479aed-ec13-4fcc-b119-8a07a523608b-plugins-config-volume\") pod \"a7479aed-ec13-4fcc-b119-8a07a523608b\" (UID: \"a7479aed-ec13-4fcc-b119-8a07a523608b\") "
Oct 29 22:59:04.567971 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.567575    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7479aed-ec13-4fcc-b119-8a07a523608b-plugins-config-volume" pod "a7479aed-ec13-4fcc-b119-8a07a523608b" (UID: "a7479aed-ec13-4fcc-b119-8a07a523608b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:04.568992 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.568970    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7479aed-ec13-4fcc-b119-8a07a523608b-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.569871 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.569837    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7479aed-ec13-4fcc-b119-8a07a523608b-kube-api-access-j49pl" pod "a7479aed-ec13-4fcc-b119-8a07a523608b" (UID: "a7479aed-ec13-4fcc-b119-8a07a523608b"). InnerVolumeSpecName "kube-api-access-j49pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:04.585389 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.585357    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"
Oct 29 22:59:04.585594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.585358    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl" event={"ID":"a75464c2-c1ca-4991-8634-7c1f417400b5","Type":"ContainerDied","Data":"1cd7ee959bc9340750566d3358d3895035c12de40f116584af98be73086e68ff"}
Oct 29 22:59:04.585594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.585513    4559 scope.go:122] "RemoveContainer" containerID="a40cf86174a72fa8d8a2faf133a965e12984d0ccfb73cc8838052f1fbe086487"
Oct 29 22:59:04.587327 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.587298    4559 generic.go:356] "Generic (PLEG): container finished" podID="d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8" containerID="92c6f0a056b33ba02c27a2f4a53abc2ab506fe6d1cad9b46e5667a3bc1ccaf27" exitCode=0
Oct 29 22:59:04.587471 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.587346    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"
Oct 29 22:59:04.587552 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.587504    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj" event={"ID":"d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8","Type":"ContainerDied","Data":"92c6f0a056b33ba02c27a2f4a53abc2ab506fe6d1cad9b46e5667a3bc1ccaf27"}
Oct 29 22:59:04.587619 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.587579    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj" event={"ID":"d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8","Type":"ContainerDied","Data":"11bcb75a194eb5f02111078e20d35e4d72cf22bcba24fb1c96583eb271b35bef"}
Oct 29 22:59:04.589707 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.589682    4559 generic.go:356] "Generic (PLEG): container finished" podID="a7479aed-ec13-4fcc-b119-8a07a523608b" containerID="79dabdefdfd5ef04d978485579ba15e7ea9ac4bae8e52f22c214d48a056bdfc3" exitCode=0
Oct 29 22:59:04.589856 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.589769    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8" event={"ID":"a7479aed-ec13-4fcc-b119-8a07a523608b","Type":"ContainerDied","Data":"79dabdefdfd5ef04d978485579ba15e7ea9ac4bae8e52f22c214d48a056bdfc3"}
Oct 29 22:59:04.589960 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.589883    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8" event={"ID":"a7479aed-ec13-4fcc-b119-8a07a523608b","Type":"ContainerDied","Data":"a63f9b4e68bda36ce38a93eb656b81ca6765a0a1a8200f5c55de3b1d273b7115"}
Oct 29 22:59:04.589960 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.589894    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"
Oct 29 22:59:04.599391 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.599369    4559 scope.go:122] "RemoveContainer" containerID="92c6f0a056b33ba02c27a2f4a53abc2ab506fe6d1cad9b46e5667a3bc1ccaf27"
Oct 29 22:59:04.613628 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.613602    4559 scope.go:122] "RemoveContainer" containerID="92c6f0a056b33ba02c27a2f4a53abc2ab506fe6d1cad9b46e5667a3bc1ccaf27"
Oct 29 22:59:04.614016 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:59:04.613992    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c6f0a056b33ba02c27a2f4a53abc2ab506fe6d1cad9b46e5667a3bc1ccaf27\": container with ID starting with 92c6f0a056b33ba02c27a2f4a53abc2ab506fe6d1cad9b46e5667a3bc1ccaf27 not found: ID does not exist" containerID="92c6f0a056b33ba02c27a2f4a53abc2ab506fe6d1cad9b46e5667a3bc1ccaf27"
Oct 29 22:59:04.614130 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.614024    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c6f0a056b33ba02c27a2f4a53abc2ab506fe6d1cad9b46e5667a3bc1ccaf27"} err="failed to get container status \"92c6f0a056b33ba02c27a2f4a53abc2ab506fe6d1cad9b46e5667a3bc1ccaf27\": rpc error: code = NotFound desc = could not find container \"92c6f0a056b33ba02c27a2f4a53abc2ab506fe6d1cad9b46e5667a3bc1ccaf27\": container with ID starting with 92c6f0a056b33ba02c27a2f4a53abc2ab506fe6d1cad9b46e5667a3bc1ccaf27 not found: ID does not exist"
Oct 29 22:59:04.614130 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.614042    4559 scope.go:122] "RemoveContainer" containerID="79dabdefdfd5ef04d978485579ba15e7ea9ac4bae8e52f22c214d48a056bdfc3"
Oct 29 22:59:04.625255 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.625229    4559 scope.go:122] "RemoveContainer" containerID="79dabdefdfd5ef04d978485579ba15e7ea9ac4bae8e52f22c214d48a056bdfc3"
Oct 29 22:59:04.625573 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:59:04.625545    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79dabdefdfd5ef04d978485579ba15e7ea9ac4bae8e52f22c214d48a056bdfc3\": container with ID starting with 79dabdefdfd5ef04d978485579ba15e7ea9ac4bae8e52f22c214d48a056bdfc3 not found: ID does not exist" containerID="79dabdefdfd5ef04d978485579ba15e7ea9ac4bae8e52f22c214d48a056bdfc3"
Oct 29 22:59:04.625651 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.625583    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79dabdefdfd5ef04d978485579ba15e7ea9ac4bae8e52f22c214d48a056bdfc3"} err="failed to get container status \"79dabdefdfd5ef04d978485579ba15e7ea9ac4bae8e52f22c214d48a056bdfc3\": rpc error: code = NotFound desc = could not find container \"79dabdefdfd5ef04d978485579ba15e7ea9ac4bae8e52f22c214d48a056bdfc3\": container with ID starting with 79dabdefdfd5ef04d978485579ba15e7ea9ac4bae8e52f22c214d48a056bdfc3 not found: ID does not exist"
Oct 29 22:59:04.670151 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.670112    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8-kube-api-access-4lpm6\" (UniqueName: \"kubernetes.io/projected/d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8-kube-api-access-4lpm6\") pod \"d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8\" (UID: \"d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8\") "
Oct 29 22:59:04.670371 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.670323    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8-plugins-config-volume\") pod \"d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8\" (UID: \"d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8\") "
Oct 29 22:59:04.670854 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.670784    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8-plugins-config-volume" pod "d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8" (UID: "d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:04.671570 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.671552    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j49pl\" (UniqueName: \"kubernetes.io/projected/a7479aed-ec13-4fcc-b119-8a07a523608b-kube-api-access-j49pl\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.671735 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.671720    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.672846 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.672803    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8-kube-api-access-4lpm6" pod "d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8" (UID: "d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8"). InnerVolumeSpecName "kube-api-access-4lpm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:04.685006 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.684969    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"]
Oct 29 22:59:04.699438 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.699411    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4796a9-58585d8495-rt4r8"]
Oct 29 22:59:04.755547 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.755509    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"]
Oct 29 22:59:04.761311 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.761278    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-06a95f-5c4c5c864d-f5xsl"]
Oct 29 22:59:04.773673 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.773636    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lpm6\" (UniqueName: \"kubernetes.io/projected/d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8-kube-api-access-4lpm6\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:04.935749 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.935658    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"]
Oct 29 22:59:04.941096 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:04.941067    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4796a9-7fc7d467d8-nkkxj"]
Oct 29 22:59:05.416298 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:05.416254    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7479aed-ec13-4fcc-b119-8a07a523608b" path="/var/lib/kubelet/pods/a7479aed-ec13-4fcc-b119-8a07a523608b/volumes"
Oct 29 22:59:05.417541 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:05.417499    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a75464c2-c1ca-4991-8634-7c1f417400b5" path="/var/lib/kubelet/pods/a75464c2-c1ca-4991-8634-7c1f417400b5/volumes"
Oct 29 22:59:05.418408 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:05.418147    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8" path="/var/lib/kubelet/pods/d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8/volumes"
Oct 29 22:59:06.143561 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:06.143524    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"
Oct 29 22:59:08.191019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:08.190987    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"
Oct 29 22:59:08.191562 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:08.191069    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"
Oct 29 22:59:10.218055 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:10.218009    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"]
Oct 29 22:59:10.218675 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:10.218326    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh" podUID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" containerName="istio-proxy" containerID="cri-o://bf730e11dd715bae70d568b0c1bb8b539302e665031e0e919f618f7a0bc79880" gracePeriod=30
Oct 29 22:59:11.125647 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:11.125612    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:59:11.125884 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:11.125690    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh" podUID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:59:12.771251 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:12.771217    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"]
Oct 29 22:59:12.771933 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:12.771476    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv" podUID="ea391d71-881c-41fb-bb0d-ee9d03a681d0" containerName="istio-proxy" containerID="cri-o://2fa78c948318a3cb204f1871757525be9943864c59acee97bca4f15f686c7f23" gracePeriod=30
Oct 29 22:59:13.632403 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.632359    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"]
Oct 29 22:59:13.635028 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.634990    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a75464c2-c1ca-4991-8634-7c1f417400b5" containerName="maas-api"
Oct 29 22:59:13.635361 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.635340    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75464c2-c1ca-4991-8634-7c1f417400b5" containerName="maas-api"
Oct 29 22:59:13.635502 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.635489    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a7479aed-ec13-4fcc-b119-8a07a523608b" containerName="payload-pre-processing"
Oct 29 22:59:13.656210 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.656179    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7479aed-ec13-4fcc-b119-8a07a523608b" containerName="payload-pre-processing"
Oct 29 22:59:13.656729 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.656707    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8" containerName="payload-processing"
Oct 29 22:59:13.656960 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.656942    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8" containerName="payload-processing"
Oct 29 22:59:13.657428 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.657409    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89cfeb6-c1a8-478a-859a-f3d2dbcbfbf8" containerName="payload-processing"
Oct 29 22:59:13.657585 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.657573    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75464c2-c1ca-4991-8634-7c1f417400b5" containerName="maas-api"
Oct 29 22:59:13.657715 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.657701    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7479aed-ec13-4fcc-b119-8a07a523608b" containerName="payload-pre-processing"
Oct 29 22:59:13.676124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.674891    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"]
Oct 29 22:59:13.676124 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.674996    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"
Oct 29 22:59:13.707250 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.707213    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:59:13.707441 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.707292    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:59:13.712978 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:59:13.712946    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:59:13.751476 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.751440    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22fxq\" (UniqueName: \"kubernetes.io/projected/581e0652-e8ab-402a-b390-acf4906d786b-kube-api-access-22fxq\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs\" (UID: \"581e0652-e8ab-402a-b390-acf4906d786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"
Oct 29 22:59:13.751714 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.751509    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581e0652-e8ab-402a-b390-acf4906d786b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs\" (UID: \"581e0652-e8ab-402a-b390-acf4906d786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"
Oct 29 22:59:13.751869 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.751844    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581e0652-e8ab-402a-b390-acf4906d786b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs\" (UID: \"581e0652-e8ab-402a-b390-acf4906d786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"
Oct 29 22:59:13.853553 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.853519    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581e0652-e8ab-402a-b390-acf4906d786b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs\" (UID: \"581e0652-e8ab-402a-b390-acf4906d786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"
Oct 29 22:59:13.854132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.853700    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22fxq\" (UniqueName: \"kubernetes.io/projected/581e0652-e8ab-402a-b390-acf4906d786b-kube-api-access-22fxq\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs\" (UID: \"581e0652-e8ab-402a-b390-acf4906d786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"
Oct 29 22:59:13.854132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.853787    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581e0652-e8ab-402a-b390-acf4906d786b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs\" (UID: \"581e0652-e8ab-402a-b390-acf4906d786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"
Oct 29 22:59:13.854628 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.854607    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581e0652-e8ab-402a-b390-acf4906d786b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs\" (UID: \"581e0652-e8ab-402a-b390-acf4906d786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"
Oct 29 22:59:13.856659 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.856635    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581e0652-e8ab-402a-b390-acf4906d786b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs\" (UID: \"581e0652-e8ab-402a-b390-acf4906d786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"
Oct 29 22:59:13.874746 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:13.874710    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22fxq\" (UniqueName: \"kubernetes.io/projected/581e0652-e8ab-402a-b390-acf4906d786b-kube-api-access-22fxq\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs\" (UID: \"581e0652-e8ab-402a-b390-acf4906d786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"
Oct 29 22:59:14.016048 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:14.016009    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"
Oct 29 22:59:14.345799 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:59:14.345771    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581e0652_e8ab_402a_b390_acf4906d786b.slice/crio-98ffc1aea4938c18bdfce333c7c0cc1ac8b551623882d8d00f96af2a4f70ea66 WatchSource:0}: Error finding container 98ffc1aea4938c18bdfce333c7c0cc1ac8b551623882d8d00f96af2a4f70ea66: Status 404 returned error can't find the container with id 98ffc1aea4938c18bdfce333c7c0cc1ac8b551623882d8d00f96af2a4f70ea66
Oct 29 22:59:14.347133 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:14.347102    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"]
Oct 29 22:59:14.793711 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:14.793675    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs" event={"ID":"581e0652-e8ab-402a-b390-acf4906d786b","Type":"ContainerStarted","Data":"c1ab77e66be41dda47cbdce4559e68bc86a434c747279712709b1e58e93af613"}
Oct 29 22:59:14.793969 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:14.793759    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs" event={"ID":"581e0652-e8ab-402a-b390-acf4906d786b","Type":"ContainerStarted","Data":"98ffc1aea4938c18bdfce333c7c0cc1ac8b551623882d8d00f96af2a4f70ea66"}
Oct 29 22:59:14.853678 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:14.853612    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs" podStartSLOduration=1.8535948389999999 podStartE2EDuration="1.853594839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:14.851722282 +0000 UTC m=+2693.763165864" watchObservedRunningTime="2026-10-29 22:59:14.853594839 +0000 UTC m=+2693.765038395"
Oct 29 22:59:15.442843 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.441917    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"]
Oct 29 22:59:15.465470 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.465096    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"
Oct 29 22:59:15.497923 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.494343    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b8f54c\""
Oct 29 22:59:15.604464 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.584184    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b8f54c-dockercfg-vzlhd\""
Oct 29 22:59:15.604464 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.598029    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"]
Oct 29 22:59:15.630057 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.605209    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"]
Oct 29 22:59:15.674055 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.661733    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"
Oct 29 22:59:15.686075 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.684307    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"]
Oct 29 22:59:15.705868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.693053    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab0aace4-e648-47e5-b31e-2203a97f5464-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5\" (UID: \"ab0aace4-e648-47e5-b31e-2203a97f5464\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"
Oct 29 22:59:15.705868 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.693187    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfhr5\" (UniqueName: \"kubernetes.io/projected/ab0aace4-e648-47e5-b31e-2203a97f5464-kube-api-access-mfhr5\") pod \"payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5\" (UID: \"ab0aace4-e648-47e5-b31e-2203a97f5464\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"
Oct 29 22:59:15.868336 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.849443    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkvlz\" (UniqueName: \"kubernetes.io/projected/ce60109b-2f7b-462b-ac57-9691a2e6b6fc-kube-api-access-lkvlz\") pod \"payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p\" (UID: \"ce60109b-2f7b-462b-ac57-9691a2e6b6fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"
Oct 29 22:59:15.868336 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.849568    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab0aace4-e648-47e5-b31e-2203a97f5464-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5\" (UID: \"ab0aace4-e648-47e5-b31e-2203a97f5464\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"
Oct 29 22:59:15.868336 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.849670    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfhr5\" (UniqueName: \"kubernetes.io/projected/ab0aace4-e648-47e5-b31e-2203a97f5464-kube-api-access-mfhr5\") pod \"payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5\" (UID: \"ab0aace4-e648-47e5-b31e-2203a97f5464\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"
Oct 29 22:59:15.868336 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.849715    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce60109b-2f7b-462b-ac57-9691a2e6b6fc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p\" (UID: \"ce60109b-2f7b-462b-ac57-9691a2e6b6fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"
Oct 29 22:59:16.002637 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.937961    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab0aace4-e648-47e5-b31e-2203a97f5464-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5\" (UID: \"ab0aace4-e648-47e5-b31e-2203a97f5464\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"
Oct 29 22:59:16.002637 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.984901    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkvlz\" (UniqueName: \"kubernetes.io/projected/ce60109b-2f7b-462b-ac57-9691a2e6b6fc-kube-api-access-lkvlz\") pod \"payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p\" (UID: \"ce60109b-2f7b-462b-ac57-9691a2e6b6fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"
Oct 29 22:59:16.002637 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.985185    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce60109b-2f7b-462b-ac57-9691a2e6b6fc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p\" (UID: \"ce60109b-2f7b-462b-ac57-9691a2e6b6fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"
Oct 29 22:59:16.002637 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:15.986184    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce60109b-2f7b-462b-ac57-9691a2e6b6fc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p\" (UID: \"ce60109b-2f7b-462b-ac57-9691a2e6b6fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"
Oct 29 22:59:16.219026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.195681    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"]
Oct 29 22:59:16.219026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.204394    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkvlz\" (UniqueName: \"kubernetes.io/projected/ce60109b-2f7b-462b-ac57-9691a2e6b6fc-kube-api-access-lkvlz\") pod \"payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p\" (UID: \"ce60109b-2f7b-462b-ac57-9691a2e6b6fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"
Oct 29 22:59:16.219026 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.217344    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.263894 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.245684    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-75b68d-openshift-default-dockercfg-9854j\""
Oct 29 22:59:16.291830 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.291778    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-token\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.292004 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.291925    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-credential-socket\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.292004 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.291960    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z8wk\" (UniqueName: \"kubernetes.io/projected/6f118086-bc70-4d6d-874a-12ebec737d0d-kube-api-access-6z8wk\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.292004 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.291999    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-data\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.292235 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.292134    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-workload-socket\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.292299 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.292236    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-podinfo\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.292299 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.292268    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-workload-certs\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.292432 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.292316    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-envoy\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.292432 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.292406    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f118086-bc70-4d6d-874a-12ebec737d0d-istiod-ca-cert\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.293801 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.293764    4559 generic.go:356] "Generic (PLEG): container finished" podID="581e0652-e8ab-402a-b390-acf4906d786b" containerID="c1ab77e66be41dda47cbdce4559e68bc86a434c747279712709b1e58e93af613" exitCode=0
Oct 29 22:59:16.294147 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.294025    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs" event={"ID":"581e0652-e8ab-402a-b390-acf4906d786b","Type":"ContainerDied","Data":"c1ab77e66be41dda47cbdce4559e68bc86a434c747279712709b1e58e93af613"}
Oct 29 22:59:16.294916 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.294890    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"]
Oct 29 22:59:16.296601 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.296316    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfhr5\" (UniqueName: \"kubernetes.io/projected/ab0aace4-e648-47e5-b31e-2203a97f5464-kube-api-access-mfhr5\") pod \"payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5\" (UID: \"ab0aace4-e648-47e5-b31e-2203a97f5464\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"
Oct 29 22:59:16.297499 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.297453    4559 generic.go:356] "Generic (PLEG): container finished" podID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" containerID="bf730e11dd715bae70d568b0c1bb8b539302e665031e0e919f618f7a0bc79880" exitCode=0
Oct 29 22:59:16.297776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.297536    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh" event={"ID":"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c","Type":"ContainerDied","Data":"bf730e11dd715bae70d568b0c1bb8b539302e665031e0e919f618f7a0bc79880"}
Oct 29 22:59:16.298004 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.297988    4559 scope.go:122] "RemoveContainer" containerID="ee567f4816a59d997f20a6010840ceabcf7bd1ac8e50bb6739a48263fb292279"
Oct 29 22:59:16.298970 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.298950    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"
Oct 29 22:59:16.451941 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.451887    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-workload-socket\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.452461 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.452430    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-podinfo\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.452629 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.452484    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-workload-certs\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.452629 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.452542    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-envoy\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.452629 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.452565    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-workload-socket\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.452863 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.452675    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f118086-bc70-4d6d-874a-12ebec737d0d-istiod-ca-cert\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.452863 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.452842    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-token\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.453059 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.453039    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-credential-socket\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.453131 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.453072    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8wk\" (UniqueName: \"kubernetes.io/projected/6f118086-bc70-4d6d-874a-12ebec737d0d-kube-api-access-6z8wk\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.453200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.453139    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-data\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.453317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.453295    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-workload-certs\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.456076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.454902    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-data\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.456076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.454931    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f118086-bc70-4d6d-874a-12ebec737d0d-istiod-ca-cert\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.456076 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.455170    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-credential-socket\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.461880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.461848    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-podinfo\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.462533 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.462502    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-envoy\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.485754 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.485631    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-token\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.487200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.487177    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8wk\" (UniqueName: \"kubernetes.io/projected/6f118086-bc70-4d6d-874a-12ebec737d0d-kube-api-access-6z8wk\") pod \"e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") " pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.531654 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.531618    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:59:16.554678 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.554643    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-workload-socket\") pod \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") "
Oct 29 22:59:16.554877 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.554747    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-data\") pod \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") "
Oct 29 22:59:16.554934 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.554900    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-kube-api-access-bw7hc\" (UniqueName: \"kubernetes.io/projected/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-kube-api-access-bw7hc\") pod \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") "
Oct 29 22:59:16.554991 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.554939    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-workload-socket" pod "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" (UID: "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:16.554991 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.554975    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-credential-socket\") pod \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") "
Oct 29 22:59:16.555086 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.555053    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istiod-ca-cert\") pod \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") "
Oct 29 22:59:16.555086 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.555066    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-data" pod "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" (UID: "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:16.555194 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.555089    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-workload-certs\") pod \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") "
Oct 29 22:59:16.555194 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.555129    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-podinfo\") pod \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") "
Oct 29 22:59:16.555321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.555219    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-credential-socket" pod "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" (UID: "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:16.555321 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.555296    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-envoy\") pod \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") "
Oct 29 22:59:16.555439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.555323    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-token\" (UniqueName: \"kubernetes.io/projected/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-token\") pod \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\" (UID: \"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c\") "
Oct 29 22:59:16.555439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.555386    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istiod-ca-cert" pod "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" (UID: "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:16.555439 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.555414    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-workload-certs" pod "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" (UID: "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:16.556335 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.556309    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.556469 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.556339    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.556469 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.556354    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.556469 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.556368    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.556469 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.556381    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.558568 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.558544    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-token" pod "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" (UID: "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:16.559217 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.559194    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-envoy" pod "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" (UID: "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:16.559860 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.559840    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-podinfo" pod "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" (UID: "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:16.560479 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.560454    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-kube-api-access-bw7hc" pod "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" (UID: "244c0d1f-3ddb-423b-9cc8-bde3dbd0064c"). InnerVolumeSpecName "kube-api-access-bw7hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:16.592095 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.592053    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"
Oct 29 22:59:16.659849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.659756    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.659849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.659786    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.659849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.659799    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.659849 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.659845    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw7hc\" (UniqueName: \"kubernetes.io/projected/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c-kube-api-access-bw7hc\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.774498 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.774414    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:16.778603 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:59:16.778554    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce60109b_2f7b_462b_ac57_9691a2e6b6fc.slice/crio-30a732f8a2a891d0ddab355a2ef4beedc3fb22ee8d6322442296aa949464923a WatchSource:0}: Error finding container 30a732f8a2a891d0ddab355a2ef4beedc3fb22ee8d6322442296aa949464923a: Status 404 returned error can't find the container with id 30a732f8a2a891d0ddab355a2ef4beedc3fb22ee8d6322442296aa949464923a
Oct 29 22:59:16.778895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.778748    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"]
Oct 29 22:59:16.937966 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:16.937933    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"]
Oct 29 22:59:16.938446 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:59:16.938421    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0aace4_e648_47e5_b31e_2203a97f5464.slice/crio-ba999a42bdd697ef69723966270c011cebb75a18d2cf116aab03b404ccb323c0 WatchSource:0}: Error finding container ba999a42bdd697ef69723966270c011cebb75a18d2cf116aab03b404ccb323c0: Status 404 returned error can't find the container with id ba999a42bdd697ef69723966270c011cebb75a18d2cf116aab03b404ccb323c0
Oct 29 22:59:17.116230 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:59:17.116197    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f118086_bc70_4d6d_874a_12ebec737d0d.slice/crio-17060c2b36cf73b164e124a79d165e5a5b1b0532223cc0d6e2f8004b49748737 WatchSource:0}: Error finding container 17060c2b36cf73b164e124a79d165e5a5b1b0532223cc0d6e2f8004b49748737: Status 404 returned error can't find the container with id 17060c2b36cf73b164e124a79d165e5a5b1b0532223cc0d6e2f8004b49748737
Oct 29 22:59:17.116709 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.116681    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"]
Oct 29 22:59:17.119609 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.119578    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:59:17.119691 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.119648    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:59:17.119691 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.119676    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:59:17.337893 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.337749    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn" event={"ID":"6f118086-bc70-4d6d-874a-12ebec737d0d","Type":"ContainerStarted","Data":"9c9dd15fc4007ca1dc5545302ff96155c2433d00839fb6e349563d7097229e61"}
Oct 29 22:59:17.338078 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.337925    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn" event={"ID":"6f118086-bc70-4d6d-874a-12ebec737d0d","Type":"ContainerStarted","Data":"17060c2b36cf73b164e124a79d165e5a5b1b0532223cc0d6e2f8004b49748737"}
Oct 29 22:59:17.344300 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.344265    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh" event={"ID":"244c0d1f-3ddb-423b-9cc8-bde3dbd0064c","Type":"ContainerDied","Data":"aa52ad6be59315adaf00034584a3a20eae85010dc6457acf2953c77c40d91354"}
Oct 29 22:59:17.344418 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.344334    4559 scope.go:122] "RemoveContainer" containerID="bf730e11dd715bae70d568b0c1bb8b539302e665031e0e919f618f7a0bc79880"
Oct 29 22:59:17.344552 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.344534    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"
Oct 29 22:59:17.348964 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.348614    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p" event={"ID":"ce60109b-2f7b-462b-ac57-9691a2e6b6fc","Type":"ContainerStarted","Data":"cdfdbab14ace0abe8e1fe05c3a925262dd456c66c64045b1556718680bdccba2"}
Oct 29 22:59:17.349166 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.349149    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"
Oct 29 22:59:17.349494 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.349475    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p" event={"ID":"ce60109b-2f7b-462b-ac57-9691a2e6b6fc","Type":"ContainerStarted","Data":"30a732f8a2a891d0ddab355a2ef4beedc3fb22ee8d6322442296aa949464923a"}
Oct 29 22:59:17.355870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.353835    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5" event={"ID":"ab0aace4-e648-47e5-b31e-2203a97f5464","Type":"ContainerStarted","Data":"f8866a8baef8247ae23212cae6b1f8ab3e6b70ccd914d484d6b7fcecd82a8fb7"}
Oct 29 22:59:17.355870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.353885    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5" event={"ID":"ab0aace4-e648-47e5-b31e-2203a97f5464","Type":"ContainerStarted","Data":"ba999a42bdd697ef69723966270c011cebb75a18d2cf116aab03b404ccb323c0"}
Oct 29 22:59:17.355870 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.353916    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"
Oct 29 22:59:17.392310 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.390317    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn" podStartSLOduration=2.390299464 podStartE2EDuration="2.390299464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:17.385039415 +0000 UTC m=+2696.296482970" watchObservedRunningTime="2026-10-29 22:59:17.390299464 +0000 UTC m=+2696.301743018"
Oct 29 22:59:17.435017 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.434867    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"]
Oct 29 22:59:17.439434 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.437671    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-xlqvh"]
Oct 29 22:59:17.585625 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.580641    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p" podStartSLOduration=2.580617572 podStartE2EDuration="2.580617572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:17.507076486 +0000 UTC m=+2696.418520094" watchObservedRunningTime="2026-10-29 22:59:17.580617572 +0000 UTC m=+2696.492061127"
Oct 29 22:59:17.621612 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.618220    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5" podStartSLOduration=2.618191185 podStartE2EDuration="2.618191185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:17.578486 +0000 UTC m=+2696.489929556" watchObservedRunningTime="2026-10-29 22:59:17.618191185 +0000 UTC m=+2696.529634740"
Oct 29 22:59:17.777842 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.774490    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:17.843694 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.843661    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"
Oct 29 22:59:17.926676 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.925345    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/581e0652-e8ab-402a-b390-acf4906d786b-kube-api-access-22fxq\" (UniqueName: \"kubernetes.io/projected/581e0652-e8ab-402a-b390-acf4906d786b-kube-api-access-22fxq\") pod \"581e0652-e8ab-402a-b390-acf4906d786b\" (UID: \"581e0652-e8ab-402a-b390-acf4906d786b\") "
Oct 29 22:59:17.926676 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.925431    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/581e0652-e8ab-402a-b390-acf4906d786b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581e0652-e8ab-402a-b390-acf4906d786b-maas-api-xks-ca\") pod \"581e0652-e8ab-402a-b390-acf4906d786b\" (UID: \"581e0652-e8ab-402a-b390-acf4906d786b\") "
Oct 29 22:59:17.926676 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.925759    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/581e0652-e8ab-402a-b390-acf4906d786b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581e0652-e8ab-402a-b390-acf4906d786b-maas-api-openshift-ca\") pod \"581e0652-e8ab-402a-b390-acf4906d786b\" (UID: \"581e0652-e8ab-402a-b390-acf4906d786b\") "
Oct 29 22:59:17.928415 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.928308    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/581e0652-e8ab-402a-b390-acf4906d786b-maas-api-openshift-ca" pod "581e0652-e8ab-402a-b390-acf4906d786b" (UID: "581e0652-e8ab-402a-b390-acf4906d786b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:17.941186 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.941128    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/581e0652-e8ab-402a-b390-acf4906d786b-kube-api-access-22fxq" pod "581e0652-e8ab-402a-b390-acf4906d786b" (UID: "581e0652-e8ab-402a-b390-acf4906d786b"). InnerVolumeSpecName "kube-api-access-22fxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:17.955542 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:17.955465    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/581e0652-e8ab-402a-b390-acf4906d786b-maas-api-xks-ca" pod "581e0652-e8ab-402a-b390-acf4906d786b" (UID: "581e0652-e8ab-402a-b390-acf4906d786b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:18.028836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.027546    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581e0652-e8ab-402a-b390-acf4906d786b-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:18.028836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.027577    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22fxq\" (UniqueName: \"kubernetes.io/projected/581e0652-e8ab-402a-b390-acf4906d786b-kube-api-access-22fxq\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:18.028836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.027587    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581e0652-e8ab-402a-b390-acf4906d786b-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:18.372604 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.372571    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"
Oct 29 22:59:18.372932 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.372584    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs" event={"ID":"581e0652-e8ab-402a-b390-acf4906d786b","Type":"ContainerDied","Data":"98ffc1aea4938c18bdfce333c7c0cc1ac8b551623882d8d00f96af2a4f70ea66"}
Oct 29 22:59:18.373258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.373236    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98ffc1aea4938c18bdfce333c7c0cc1ac8b551623882d8d00f96af2a4f70ea66"
Oct 29 22:59:18.378111 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.375782    4559 generic.go:356] "Generic (PLEG): container finished" podID="ea391d71-881c-41fb-bb0d-ee9d03a681d0" containerID="2fa78c948318a3cb204f1871757525be9943864c59acee97bca4f15f686c7f23" exitCode=0
Oct 29 22:59:18.378111 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.376801    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv" event={"ID":"ea391d71-881c-41fb-bb0d-ee9d03a681d0","Type":"ContainerDied","Data":"2fa78c948318a3cb204f1871757525be9943864c59acee97bca4f15f686c7f23"}
Oct 29 22:59:18.378111 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.376879    4559 scope.go:122] "RemoveContainer" containerID="7b01695e3676eae49a793a3963e36449026fef50438f5c10a7271ed37e14c966"
Oct 29 22:59:18.542883 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.542851    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:59:18.637019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.636935    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-envoy\") pod \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") "
Oct 29 22:59:18.637157 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.637073    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-workload-socket\") pod \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") "
Oct 29 22:59:18.637157 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.637135    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-token\" (UniqueName: \"kubernetes.io/projected/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-token\") pod \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") "
Oct 29 22:59:18.637242 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.637173    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-credential-socket\") pod \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") "
Oct 29 22:59:18.637242 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.637209    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea391d71-881c-41fb-bb0d-ee9d03a681d0-kube-api-access-bcd29\" (UniqueName: \"kubernetes.io/projected/ea391d71-881c-41fb-bb0d-ee9d03a681d0-kube-api-access-bcd29\") pod \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") "
Oct 29 22:59:18.637317 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.637307    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-workload-certs\") pod \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") "
Oct 29 22:59:18.637449 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.637435    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-data\") pod \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") "
Oct 29 22:59:18.637496 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.637470    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-podinfo\") pod \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") "
Oct 29 22:59:18.637530 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.637508    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istiod-ca-cert\") pod \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\" (UID: \"ea391d71-881c-41fb-bb0d-ee9d03a681d0\") "
Oct 29 22:59:18.638955 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.638927    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istiod-ca-cert" pod "ea391d71-881c-41fb-bb0d-ee9d03a681d0" (UID: "ea391d71-881c-41fb-bb0d-ee9d03a681d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:18.641629 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.641595    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-envoy" pod "ea391d71-881c-41fb-bb0d-ee9d03a681d0" (UID: "ea391d71-881c-41fb-bb0d-ee9d03a681d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:18.641854 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.641834    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-workload-certs" pod "ea391d71-881c-41fb-bb0d-ee9d03a681d0" (UID: "ea391d71-881c-41fb-bb0d-ee9d03a681d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:18.642019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.642001    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-data" pod "ea391d71-881c-41fb-bb0d-ee9d03a681d0" (UID: "ea391d71-881c-41fb-bb0d-ee9d03a681d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:18.642209 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.642193    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-workload-socket" pod "ea391d71-881c-41fb-bb0d-ee9d03a681d0" (UID: "ea391d71-881c-41fb-bb0d-ee9d03a681d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:18.642378 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.642361    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-credential-socket" pod "ea391d71-881c-41fb-bb0d-ee9d03a681d0" (UID: "ea391d71-881c-41fb-bb0d-ee9d03a681d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:18.647583 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.647550    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-podinfo" pod "ea391d71-881c-41fb-bb0d-ee9d03a681d0" (UID: "ea391d71-881c-41fb-bb0d-ee9d03a681d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:18.656405 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.655168    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-token" pod "ea391d71-881c-41fb-bb0d-ee9d03a681d0" (UID: "ea391d71-881c-41fb-bb0d-ee9d03a681d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:18.656405 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.655271    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea391d71-881c-41fb-bb0d-ee9d03a681d0-kube-api-access-bcd29" pod "ea391d71-881c-41fb-bb0d-ee9d03a681d0" (UID: "ea391d71-881c-41fb-bb0d-ee9d03a681d0"). InnerVolumeSpecName "kube-api-access-bcd29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:18.740258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.740221    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:18.740258 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.740259    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:18.740465 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.740273    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:18.740465 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.740286    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:18.740465 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.740299    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:18.740465 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.740311    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:18.740465 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.740322    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea391d71-881c-41fb-bb0d-ee9d03a681d0-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:18.740465 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.740334    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391d71-881c-41fb-bb0d-ee9d03a681d0-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:18.740465 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.740347    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcd29\" (UniqueName: \"kubernetes.io/projected/ea391d71-881c-41fb-bb0d-ee9d03a681d0-kube-api-access-bcd29\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:18.781105 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:18.781074    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:19.230251 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:19.230215    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" path="/var/lib/kubelet/pods/244c0d1f-3ddb-423b-9cc8-bde3dbd0064c/volumes"
Oct 29 22:59:19.386646 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:19.386619    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"
Oct 29 22:59:19.386913 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:19.386666    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv" event={"ID":"ea391d71-881c-41fb-bb0d-ee9d03a681d0","Type":"ContainerDied","Data":"2a64da2ea5820668f2f1bc95e46c02ada517bbd6622ab94cd464ea3b5f4a01f7"}
Oct 29 22:59:19.386913 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:19.386740    4559 scope.go:122] "RemoveContainer" containerID="2fa78c948318a3cb204f1871757525be9943864c59acee97bca4f15f686c7f23"
Oct 29 22:59:19.387303 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:19.387283    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:19.388480 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:19.388460    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 22:59:19.455233 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:19.455198    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"]
Oct 29 22:59:19.466665 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:19.466635    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4796a9-openshift-default-75cd4d45f7-gm2vv"]
Oct 29 22:59:21.362002 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:21.361954    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea391d71-881c-41fb-bb0d-ee9d03a681d0" path="/var/lib/kubelet/pods/ea391d71-881c-41fb-bb0d-ee9d03a681d0/volumes"
Oct 29 22:59:21.846101 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:21.846066    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:59:21.859090 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:21.859063    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 22:59:24.219355 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:24.219319    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"]
Oct 29 22:59:25.456834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:25.446188    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"]
Oct 29 22:59:25.456834 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:25.446443    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9" podUID="b43b52f1-aade-4064-9d8a-111d155c659b" containerName="maas-api" containerID="cri-o://19a04cc987a4b00abeb9015755d47c906f1b1925c4139b18b79758007e0212af" gracePeriod=30
Oct 29 22:59:25.998572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:25.998537    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:59:26.103447 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.103411    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b43b52f1-aade-4064-9d8a-111d155c659b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43b52f1-aade-4064-9d8a-111d155c659b-openshift-service-ca\") pod \"b43b52f1-aade-4064-9d8a-111d155c659b\" (UID: \"b43b52f1-aade-4064-9d8a-111d155c659b\") "
Oct 29 22:59:26.103447 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.103449    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b43b52f1-aade-4064-9d8a-111d155c659b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b43b52f1-aade-4064-9d8a-111d155c659b-maas-api-tls\") pod \"b43b52f1-aade-4064-9d8a-111d155c659b\" (UID: \"b43b52f1-aade-4064-9d8a-111d155c659b\") "
Oct 29 22:59:26.103716 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.103523    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b43b52f1-aade-4064-9d8a-111d155c659b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b43b52f1-aade-4064-9d8a-111d155c659b-metrics-certs\") pod \"b43b52f1-aade-4064-9d8a-111d155c659b\" (UID: \"b43b52f1-aade-4064-9d8a-111d155c659b\") "
Oct 29 22:59:26.103787 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.103725    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b43b52f1-aade-4064-9d8a-111d155c659b-kube-api-access-hzcgx\" (UniqueName: \"kubernetes.io/projected/b43b52f1-aade-4064-9d8a-111d155c659b-kube-api-access-hzcgx\") pod \"b43b52f1-aade-4064-9d8a-111d155c659b\" (UID: \"b43b52f1-aade-4064-9d8a-111d155c659b\") "
Oct 29 22:59:26.103880 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.103864    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b43b52f1-aade-4064-9d8a-111d155c659b-openshift-service-ca" pod "b43b52f1-aade-4064-9d8a-111d155c659b" (UID: "b43b52f1-aade-4064-9d8a-111d155c659b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:26.105047 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.105014    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43b52f1-aade-4064-9d8a-111d155c659b-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:26.106620 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.106575    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b43b52f1-aade-4064-9d8a-111d155c659b-kube-api-access-hzcgx" pod "b43b52f1-aade-4064-9d8a-111d155c659b" (UID: "b43b52f1-aade-4064-9d8a-111d155c659b"). InnerVolumeSpecName "kube-api-access-hzcgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:26.106912 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.106891    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b43b52f1-aade-4064-9d8a-111d155c659b-maas-api-tls" pod "b43b52f1-aade-4064-9d8a-111d155c659b" (UID: "b43b52f1-aade-4064-9d8a-111d155c659b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:26.106997 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.106916    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b43b52f1-aade-4064-9d8a-111d155c659b-metrics-certs" pod "b43b52f1-aade-4064-9d8a-111d155c659b" (UID: "b43b52f1-aade-4064-9d8a-111d155c659b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:26.111525 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.111480    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"]
Oct 29 22:59:26.112397 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.112376    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ea391d71-881c-41fb-bb0d-ee9d03a681d0" containerName="istio-proxy"
Oct 29 22:59:26.112397 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.112396    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea391d71-881c-41fb-bb0d-ee9d03a681d0" containerName="istio-proxy"
Oct 29 22:59:26.112572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.112439    4559 cpu_manager.go:414] "containerMap: removing container" podUID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" containerName="istio-proxy"
Oct 29 22:59:26.112572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.112444    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" containerName="istio-proxy"
Oct 29 22:59:26.112572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.112454    4559 cpu_manager.go:414] "containerMap: removing container" podUID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" containerName="istio-proxy"
Oct 29 22:59:26.112572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.112458    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" containerName="istio-proxy"
Oct 29 22:59:26.112572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.112478    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b43b52f1-aade-4064-9d8a-111d155c659b" containerName="maas-api"
Oct 29 22:59:26.112572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.112486    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b43b52f1-aade-4064-9d8a-111d155c659b" containerName="maas-api"
Oct 29 22:59:26.112572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.112500    4559 cpu_manager.go:414] "containerMap: removing container" podUID="581e0652-e8ab-402a-b390-acf4906d786b" containerName="revoke-keys"
Oct 29 22:59:26.112572 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.112504    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="581e0652-e8ab-402a-b390-acf4906d786b" containerName="revoke-keys"
Oct 29 22:59:26.113132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.112614    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" containerName="istio-proxy"
Oct 29 22:59:26.113132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.112625    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="244c0d1f-3ddb-423b-9cc8-bde3dbd0064c" containerName="istio-proxy"
Oct 29 22:59:26.113132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.112634    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b43b52f1-aade-4064-9d8a-111d155c659b" containerName="maas-api"
Oct 29 22:59:26.113132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.112642    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="581e0652-e8ab-402a-b390-acf4906d786b" containerName="revoke-keys"
Oct 29 22:59:26.113132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.112649    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea391d71-881c-41fb-bb0d-ee9d03a681d0" containerName="istio-proxy"
Oct 29 22:59:26.113132 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.112656    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea391d71-881c-41fb-bb0d-ee9d03a681d0" containerName="istio-proxy"
Oct 29 22:59:26.116829 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.116794    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"
Oct 29 22:59:26.142352 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.142290    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-75b68d\""
Oct 29 22:59:26.143008 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.142982    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-75b68d-dockercfg-tjvpg\""
Oct 29 22:59:26.148354 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.146677    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"]
Oct 29 22:59:26.158945 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.158500    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"]
Oct 29 22:59:26.159978 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.159923    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ea391d71-881c-41fb-bb0d-ee9d03a681d0" containerName="istio-proxy"
Oct 29 22:59:26.159978 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.159953    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea391d71-881c-41fb-bb0d-ee9d03a681d0" containerName="istio-proxy"
Oct 29 22:59:26.165290 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.165259    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"
Oct 29 22:59:26.194755 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.194721    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"]
Oct 29 22:59:26.207019 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.206976    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6s88\" (UniqueName: \"kubernetes.io/projected/c2822e0d-87da-4882-a4ed-acebabc7cc8b-kube-api-access-r6s88\") pod \"payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt\" (UID: \"c2822e0d-87da-4882-a4ed-acebabc7cc8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"
Oct 29 22:59:26.207238 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.207181    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-854gr\" (UniqueName: \"kubernetes.io/projected/370fbc49-ac99-41e3-9375-098433ebd4da-kube-api-access-854gr\") pod \"payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh\" (UID: \"370fbc49-ac99-41e3-9375-098433ebd4da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"
Oct 29 22:59:26.207238 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.207220    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370fbc49-ac99-41e3-9375-098433ebd4da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh\" (UID: \"370fbc49-ac99-41e3-9375-098433ebd4da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"
Oct 29 22:59:26.207381 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.207364    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2822e0d-87da-4882-a4ed-acebabc7cc8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt\" (UID: \"c2822e0d-87da-4882-a4ed-acebabc7cc8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"
Oct 29 22:59:26.207719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.207650    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b43b52f1-aade-4064-9d8a-111d155c659b-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:26.207719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.207677    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b43b52f1-aade-4064-9d8a-111d155c659b-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:26.207719 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.207691    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzcgx\" (UniqueName: \"kubernetes.io/projected/b43b52f1-aade-4064-9d8a-111d155c659b-kube-api-access-hzcgx\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:26.309690 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.309647    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-854gr\" (UniqueName: \"kubernetes.io/projected/370fbc49-ac99-41e3-9375-098433ebd4da-kube-api-access-854gr\") pod \"payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh\" (UID: \"370fbc49-ac99-41e3-9375-098433ebd4da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"
Oct 29 22:59:26.309690 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.309692    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370fbc49-ac99-41e3-9375-098433ebd4da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh\" (UID: \"370fbc49-ac99-41e3-9375-098433ebd4da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"
Oct 29 22:59:26.309983 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.309800    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2822e0d-87da-4882-a4ed-acebabc7cc8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt\" (UID: \"c2822e0d-87da-4882-a4ed-acebabc7cc8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"
Oct 29 22:59:26.310074 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.310057    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6s88\" (UniqueName: \"kubernetes.io/projected/c2822e0d-87da-4882-a4ed-acebabc7cc8b-kube-api-access-r6s88\") pod \"payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt\" (UID: \"c2822e0d-87da-4882-a4ed-acebabc7cc8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"
Oct 29 22:59:26.310565 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.310504    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370fbc49-ac99-41e3-9375-098433ebd4da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh\" (UID: \"370fbc49-ac99-41e3-9375-098433ebd4da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"
Oct 29 22:59:26.310800 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.310778    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2822e0d-87da-4882-a4ed-acebabc7cc8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt\" (UID: \"c2822e0d-87da-4882-a4ed-acebabc7cc8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"
Oct 29 22:59:26.391836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.376558    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-854gr\" (UniqueName: \"kubernetes.io/projected/370fbc49-ac99-41e3-9375-098433ebd4da-kube-api-access-854gr\") pod \"payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh\" (UID: \"370fbc49-ac99-41e3-9375-098433ebd4da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"
Oct 29 22:59:26.391836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.391564    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6s88\" (UniqueName: \"kubernetes.io/projected/c2822e0d-87da-4882-a4ed-acebabc7cc8b-kube-api-access-r6s88\") pod \"payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt\" (UID: \"c2822e0d-87da-4882-a4ed-acebabc7cc8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"
Oct 29 22:59:26.441884 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.436200    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"
Oct 29 22:59:26.507340 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.507300    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"
Oct 29 22:59:26.712180 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.712090    4559 generic.go:356] "Generic (PLEG): container finished" podID="b43b52f1-aade-4064-9d8a-111d155c659b" containerID="19a04cc987a4b00abeb9015755d47c906f1b1925c4139b18b79758007e0212af" exitCode=0
Oct 29 22:59:26.712693 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.712670    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9" event={"ID":"b43b52f1-aade-4064-9d8a-111d155c659b","Type":"ContainerDied","Data":"19a04cc987a4b00abeb9015755d47c906f1b1925c4139b18b79758007e0212af"}
Oct 29 22:59:26.712873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.712852    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9" event={"ID":"b43b52f1-aade-4064-9d8a-111d155c659b","Type":"ContainerDied","Data":"262a22edc6c2f2156a131dbec3239f1c6535f66ec52c5dd51917a28bd66c1391"}
Oct 29 22:59:26.712991 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.712978    4559 scope.go:122] "RemoveContainer" containerID="19a04cc987a4b00abeb9015755d47c906f1b1925c4139b18b79758007e0212af"
Oct 29 22:59:26.713271 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.713257    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"
Oct 29 22:59:26.778510 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.776636    4559 scope.go:122] "RemoveContainer" containerID="19a04cc987a4b00abeb9015755d47c906f1b1925c4139b18b79758007e0212af"
Oct 29 22:59:26.778510 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:59:26.777400    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19a04cc987a4b00abeb9015755d47c906f1b1925c4139b18b79758007e0212af\": container with ID starting with 19a04cc987a4b00abeb9015755d47c906f1b1925c4139b18b79758007e0212af not found: ID does not exist" containerID="19a04cc987a4b00abeb9015755d47c906f1b1925c4139b18b79758007e0212af"
Oct 29 22:59:26.778510 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.777438    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a04cc987a4b00abeb9015755d47c906f1b1925c4139b18b79758007e0212af"} err="failed to get container status \"19a04cc987a4b00abeb9015755d47c906f1b1925c4139b18b79758007e0212af\": rpc error: code = NotFound desc = could not find container \"19a04cc987a4b00abeb9015755d47c906f1b1925c4139b18b79758007e0212af\": container with ID starting with 19a04cc987a4b00abeb9015755d47c906f1b1925c4139b18b79758007e0212af not found: ID does not exist"
Oct 29 22:59:26.853650 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.853610    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"]
Oct 29 22:59:26.863669 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:26.863631    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-db7419-88549679c-jv4t9"]
Oct 29 22:59:27.050826 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:27.049110    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"]
Oct 29 22:59:27.050826 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:59:27.050069    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2822e0d_87da_4882_a4ed_acebabc7cc8b.slice/crio-71102278bcff5ecb77b95c71bc10a26dbc43d8029de86e426507e821a9260bd3 WatchSource:0}: Error finding container 71102278bcff5ecb77b95c71bc10a26dbc43d8029de86e426507e821a9260bd3: Status 404 returned error can't find the container with id 71102278bcff5ecb77b95c71bc10a26dbc43d8029de86e426507e821a9260bd3
Oct 29 22:59:27.069622 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:27.069588    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"]
Oct 29 22:59:27.070600 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:59:27.070572    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370fbc49_ac99_41e3_9375_098433ebd4da.slice/crio-536816625d71136de2e046f88bbae7093c1b0452528b140d61fd67a5e670fe13 WatchSource:0}: Error finding container 536816625d71136de2e046f88bbae7093c1b0452528b140d61fd67a5e670fe13: Status 404 returned error can't find the container with id 536816625d71136de2e046f88bbae7093c1b0452528b140d61fd67a5e670fe13
Oct 29 22:59:27.231750 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:27.231713    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b43b52f1-aade-4064-9d8a-111d155c659b" path="/var/lib/kubelet/pods/b43b52f1-aade-4064-9d8a-111d155c659b/volumes"
Oct 29 22:59:27.729200 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:27.729156    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt" event={"ID":"c2822e0d-87da-4882-a4ed-acebabc7cc8b","Type":"ContainerStarted","Data":"e3ceae8c9809c96c1c4edf12965e9a6f82bd420aa76ca83305d6e6665658bd4c"}
Oct 29 22:59:27.729698 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:27.729254    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"
Oct 29 22:59:27.729698 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:27.729324    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt" event={"ID":"c2822e0d-87da-4882-a4ed-acebabc7cc8b","Type":"ContainerStarted","Data":"71102278bcff5ecb77b95c71bc10a26dbc43d8029de86e426507e821a9260bd3"}
Oct 29 22:59:27.731776 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:27.731739    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh" event={"ID":"370fbc49-ac99-41e3-9375-098433ebd4da","Type":"ContainerStarted","Data":"74c9029925c67c596c49882746be99c717f68fa59f9b666e66dbd98fa7f79f99"}
Oct 29 22:59:27.731973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:27.731861    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"
Oct 29 22:59:27.731973 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:27.731904    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh" event={"ID":"370fbc49-ac99-41e3-9375-098433ebd4da","Type":"ContainerStarted","Data":"536816625d71136de2e046f88bbae7093c1b0452528b140d61fd67a5e670fe13"}
Oct 29 22:59:27.778921 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:27.778854    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt" podStartSLOduration=1.778832108 podStartE2EDuration="1.778832108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:27.772805265 +0000 UTC m=+2706.684248819" watchObservedRunningTime="2026-10-29 22:59:27.778832108 +0000 UTC m=+2706.690275669"
Oct 29 22:59:27.822192 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:27.822133    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh" podStartSLOduration=1.822117599 podStartE2EDuration="1.822117599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:27.817630452 +0000 UTC m=+2706.729074005" watchObservedRunningTime="2026-10-29 22:59:27.822117599 +0000 UTC m=+2706.733561153"
Oct 29 22:59:28.380687 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:28.380654    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"
Oct 29 22:59:28.380936 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:28.380731    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"
Oct 29 22:59:29.220459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:29.220406    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m" podUID="d838fae2-87e9-4874-bb79-92a34e503ce6" containerName="payload-pre-processing" containerID="cri-o://f302bc68b25b097a604a22004fd9d01e628990029018c4e357687848d577b27c" gracePeriod=25
Oct 29 22:59:33.747768 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:33.747715    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m" podUID="d838fae2-87e9-4874-bb79-92a34e503ce6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:59:35.354685 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:35.354644    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"]
Oct 29 22:59:35.362648 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:35.362619    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"
Oct 29 22:59:35.378298 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:35.378254    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"]
Oct 29 22:59:35.383348 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:35.383314    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:59:35.383522 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:35.383410    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:59:35.463772 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:35.463726    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74df5d6d-2367-462e-a46e-57feb80833fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4\" (UID: \"74df5d6d-2367-462e-a46e-57feb80833fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"
Oct 29 22:59:35.464025 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:35.463779    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzsvc\" (UniqueName: \"kubernetes.io/projected/74df5d6d-2367-462e-a46e-57feb80833fe-kube-api-access-dzsvc\") pod \"maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4\" (UID: \"74df5d6d-2367-462e-a46e-57feb80833fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"
Oct 29 22:59:35.464025 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:35.463918    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74df5d6d-2367-462e-a46e-57feb80833fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4\" (UID: \"74df5d6d-2367-462e-a46e-57feb80833fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"
Oct 29 22:59:35.566424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:35.566385    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74df5d6d-2367-462e-a46e-57feb80833fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4\" (UID: \"74df5d6d-2367-462e-a46e-57feb80833fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"
Oct 29 22:59:35.566665 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:35.566440    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzsvc\" (UniqueName: \"kubernetes.io/projected/74df5d6d-2367-462e-a46e-57feb80833fe-kube-api-access-dzsvc\") pod \"maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4\" (UID: \"74df5d6d-2367-462e-a46e-57feb80833fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"
Oct 29 22:59:35.566665 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:35.566472    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74df5d6d-2367-462e-a46e-57feb80833fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4\" (UID: \"74df5d6d-2367-462e-a46e-57feb80833fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"
Oct 29 22:59:35.567424 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:35.567396    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74df5d6d-2367-462e-a46e-57feb80833fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4\" (UID: \"74df5d6d-2367-462e-a46e-57feb80833fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"
Oct 29 22:59:35.569281 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:35.569261    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74df5d6d-2367-462e-a46e-57feb80833fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4\" (UID: \"74df5d6d-2367-462e-a46e-57feb80833fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"
Oct 29 22:59:35.588388 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:35.588355    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzsvc\" (UniqueName: \"kubernetes.io/projected/74df5d6d-2367-462e-a46e-57feb80833fe-kube-api-access-dzsvc\") pod \"maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4\" (UID: \"74df5d6d-2367-462e-a46e-57feb80833fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"
Oct 29 22:59:35.685554 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:35.685519    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"
Oct 29 22:59:36.157233 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:36.157195    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"]
Oct 29 22:59:36.158238 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:59:36.158205    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74df5d6d_2367_462e_a46e_57feb80833fe.slice/crio-013f81be8a8ea5ec70b3ace3e5c2e1436f6f0e83995213a9f6bd901ec5bcd258 WatchSource:0}: Error finding container 013f81be8a8ea5ec70b3ace3e5c2e1436f6f0e83995213a9f6bd901ec5bcd258: Status 404 returned error can't find the container with id 013f81be8a8ea5ec70b3ace3e5c2e1436f6f0e83995213a9f6bd901ec5bcd258
Oct 29 22:59:36.992533 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:36.992493    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4" event={"ID":"74df5d6d-2367-462e-a46e-57feb80833fe","Type":"ContainerStarted","Data":"e5576a935a4eb7ba470c96a549566b2650797d5d73eb14bc44219803d43266e8"}
Oct 29 22:59:36.993004 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:36.992547    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4" event={"ID":"74df5d6d-2367-462e-a46e-57feb80833fe","Type":"ContainerStarted","Data":"013f81be8a8ea5ec70b3ace3e5c2e1436f6f0e83995213a9f6bd901ec5bcd258"}
Oct 29 22:59:37.038155 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:37.038093    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4" podStartSLOduration=2.038074186 podStartE2EDuration="2.038074186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:37.033597975 +0000 UTC m=+2715.945041530" watchObservedRunningTime="2026-10-29 22:59:37.038074186 +0000 UTC m=+2715.949517742"
Oct 29 22:59:38.003004 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:38.002967    4559 generic.go:356] "Generic (PLEG): container finished" podID="74df5d6d-2367-462e-a46e-57feb80833fe" containerID="e5576a935a4eb7ba470c96a549566b2650797d5d73eb14bc44219803d43266e8" exitCode=0
Oct 29 22:59:38.003410 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:38.003054    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4" event={"ID":"74df5d6d-2367-462e-a46e-57feb80833fe","Type":"ContainerDied","Data":"e5576a935a4eb7ba470c96a549566b2650797d5d73eb14bc44219803d43266e8"}
Oct 29 22:59:38.797474 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:38.796002    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"
Oct 29 22:59:38.805836 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:38.804996    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"
Oct 29 22:59:39.486705 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:39.462639    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:39.486705 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:39.467774    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:59:39.784125 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:39.784094    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"
Oct 29 22:59:39.874696 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:39.874650    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74df5d6d-2367-462e-a46e-57feb80833fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74df5d6d-2367-462e-a46e-57feb80833fe-maas-api-xks-ca\") pod \"74df5d6d-2367-462e-a46e-57feb80833fe\" (UID: \"74df5d6d-2367-462e-a46e-57feb80833fe\") "
Oct 29 22:59:39.874914 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:39.874887    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74df5d6d-2367-462e-a46e-57feb80833fe-kube-api-access-dzsvc\" (UniqueName: \"kubernetes.io/projected/74df5d6d-2367-462e-a46e-57feb80833fe-kube-api-access-dzsvc\") pod \"74df5d6d-2367-462e-a46e-57feb80833fe\" (UID: \"74df5d6d-2367-462e-a46e-57feb80833fe\") "
Oct 29 22:59:39.875229 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:39.875201    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74df5d6d-2367-462e-a46e-57feb80833fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74df5d6d-2367-462e-a46e-57feb80833fe-maas-api-openshift-ca\") pod \"74df5d6d-2367-462e-a46e-57feb80833fe\" (UID: \"74df5d6d-2367-462e-a46e-57feb80833fe\") "
Oct 29 22:59:39.876594 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:39.876569    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74df5d6d-2367-462e-a46e-57feb80833fe-maas-api-openshift-ca" pod "74df5d6d-2367-462e-a46e-57feb80833fe" (UID: "74df5d6d-2367-462e-a46e-57feb80833fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:39.880064 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:39.880039    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74df5d6d-2367-462e-a46e-57feb80833fe-kube-api-access-dzsvc" pod "74df5d6d-2367-462e-a46e-57feb80833fe" (UID: "74df5d6d-2367-462e-a46e-57feb80833fe"). InnerVolumeSpecName "kube-api-access-dzsvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:39.883268 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:39.883243    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74df5d6d-2367-462e-a46e-57feb80833fe-maas-api-xks-ca" pod "74df5d6d-2367-462e-a46e-57feb80833fe" (UID: "74df5d6d-2367-462e-a46e-57feb80833fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:39.978262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:39.978156    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74df5d6d-2367-462e-a46e-57feb80833fe-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:39.978262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:39.978203    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzsvc\" (UniqueName: \"kubernetes.io/projected/74df5d6d-2367-462e-a46e-57feb80833fe-kube-api-access-dzsvc\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:39.978262 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:39.978220    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74df5d6d-2367-462e-a46e-57feb80833fe-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:40.096906 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.096873    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"]
Oct 29 22:59:40.097819 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.097793    4559 cpu_manager.go:414] "containerMap: removing container" podUID="74df5d6d-2367-462e-a46e-57feb80833fe" containerName="revoke-keys"
Oct 29 22:59:40.097967 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.097827    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74df5d6d-2367-462e-a46e-57feb80833fe" containerName="revoke-keys"
Oct 29 22:59:40.098035 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.098023    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="74df5d6d-2367-462e-a46e-57feb80833fe" containerName="revoke-keys"
Oct 29 22:59:40.102109 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.102092    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"
Oct 29 22:59:40.117874 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.117847    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-63e89c-dockercfg-dtwkg\""
Oct 29 22:59:40.119739 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.119691    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-63e89c\""
Oct 29 22:59:40.123717 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.123689    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"]
Oct 29 22:59:40.182041 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.181997    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b93002e-a2d9-4bd4-9bb3-266576873ebb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9\" (UID: \"6b93002e-a2d9-4bd4-9bb3-266576873ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"
Oct 29 22:59:40.182309 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.182281    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnbh9\" (UniqueName: \"kubernetes.io/projected/6b93002e-a2d9-4bd4-9bb3-266576873ebb-kube-api-access-gnbh9\") pod \"payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9\" (UID: \"6b93002e-a2d9-4bd4-9bb3-266576873ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"
Oct 29 22:59:40.226226 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.226193    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"]
Oct 29 22:59:40.321900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.317566    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b93002e-a2d9-4bd4-9bb3-266576873ebb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9\" (UID: \"6b93002e-a2d9-4bd4-9bb3-266576873ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"
Oct 29 22:59:40.321900 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.317857    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnbh9\" (UniqueName: \"kubernetes.io/projected/6b93002e-a2d9-4bd4-9bb3-266576873ebb-kube-api-access-gnbh9\") pod \"payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9\" (UID: \"6b93002e-a2d9-4bd4-9bb3-266576873ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"
Oct 29 22:59:40.338863 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.323823    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b93002e-a2d9-4bd4-9bb3-266576873ebb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9\" (UID: \"6b93002e-a2d9-4bd4-9bb3-266576873ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"
Oct 29 22:59:40.507850 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.475344    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"]
Oct 29 22:59:40.745838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.716136    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"]
Oct 29 22:59:40.745838 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.716258    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"
Oct 29 22:59:40.762748 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.758714    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4" event={"ID":"74df5d6d-2367-462e-a46e-57feb80833fe","Type":"ContainerDied","Data":"013f81be8a8ea5ec70b3ace3e5c2e1436f6f0e83995213a9f6bd901ec5bcd258"}
Oct 29 22:59:40.762748 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.758778    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="013f81be8a8ea5ec70b3ace3e5c2e1436f6f0e83995213a9f6bd901ec5bcd258"
Oct 29 22:59:40.762748 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.758866    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"
Oct 29 22:59:40.808895 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.808404    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnbh9\" (UniqueName: \"kubernetes.io/projected/6b93002e-a2d9-4bd4-9bb3-266576873ebb-kube-api-access-gnbh9\") pod \"payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9\" (UID: \"6b93002e-a2d9-4bd4-9bb3-266576873ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"
Oct 29 22:59:40.816698 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.816646    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwr7\" (UniqueName: \"kubernetes.io/projected/0400c09d-e610-4055-b068-c00c6f156c9c-kube-api-access-mbwr7\") pod \"payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k\" (UID: \"0400c09d-e610-4055-b068-c00c6f156c9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"
Oct 29 22:59:40.816884 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.816767    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0400c09d-e610-4055-b068-c00c6f156c9c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k\" (UID: \"0400c09d-e610-4055-b068-c00c6f156c9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"
Oct 29 22:59:40.918558 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.918516    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0400c09d-e610-4055-b068-c00c6f156c9c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k\" (UID: \"0400c09d-e610-4055-b068-c00c6f156c9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"
Oct 29 22:59:40.918851 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.918833    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwr7\" (UniqueName: \"kubernetes.io/projected/0400c09d-e610-4055-b068-c00c6f156c9c-kube-api-access-mbwr7\") pod \"payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k\" (UID: \"0400c09d-e610-4055-b068-c00c6f156c9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"
Oct 29 22:59:40.919462 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.919439    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0400c09d-e610-4055-b068-c00c6f156c9c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k\" (UID: \"0400c09d-e610-4055-b068-c00c6f156c9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"
Oct 29 22:59:40.967010 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:40.966961    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwr7\" (UniqueName: \"kubernetes.io/projected/0400c09d-e610-4055-b068-c00c6f156c9c-kube-api-access-mbwr7\") pod \"payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k\" (UID: \"0400c09d-e610-4055-b068-c00c6f156c9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"
Oct 29 22:59:41.018693 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:41.018599    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"
Oct 29 22:59:41.108086 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:41.108050    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"
Oct 29 22:59:41.377738 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:59:41.377707    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b93002e_a2d9_4bd4_9bb3_266576873ebb.slice/crio-d85d61977c1fb1e32d59b168b387f48a0e0a12119966fe9f40e7e4cc43966cc4 WatchSource:0}: Error finding container d85d61977c1fb1e32d59b168b387f48a0e0a12119966fe9f40e7e4cc43966cc4: Status 404 returned error can't find the container with id d85d61977c1fb1e32d59b168b387f48a0e0a12119966fe9f40e7e4cc43966cc4
Oct 29 22:59:41.379505 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:41.379462    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"]
Oct 29 22:59:41.484089 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:41.484061    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"]
Oct 29 22:59:41.484321 ip-10-0-138-2 kubenswrapper[4559]: W1029 22:59:41.484145    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0400c09d_e610_4055_b068_c00c6f156c9c.slice/crio-aecfe230c2d7645b1866dd63007572822f1456b422f4e7528e885265bb3e6cf6 WatchSource:0}: Error finding container aecfe230c2d7645b1866dd63007572822f1456b422f4e7528e885265bb3e6cf6: Status 404 returned error can't find the container with id aecfe230c2d7645b1866dd63007572822f1456b422f4e7528e885265bb3e6cf6
Oct 29 22:59:41.772632 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:41.772591    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k" event={"ID":"0400c09d-e610-4055-b068-c00c6f156c9c","Type":"ContainerStarted","Data":"6af1c113c396b1f24cbf5fa23c33de79b401cb398f84d0eac0dca7f990b8d8c0"}
Oct 29 22:59:41.773084 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:41.772659    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k" event={"ID":"0400c09d-e610-4055-b068-c00c6f156c9c","Type":"ContainerStarted","Data":"aecfe230c2d7645b1866dd63007572822f1456b422f4e7528e885265bb3e6cf6"}
Oct 29 22:59:41.773084 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:41.772691    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"
Oct 29 22:59:41.774828 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:41.774774    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9" event={"ID":"6b93002e-a2d9-4bd4-9bb3-266576873ebb","Type":"ContainerStarted","Data":"429f204bcaa63c9fe7ee321ba7a3a0f45d521631f2c01610992cfe9d69122593"}
Oct 29 22:59:41.774974 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:41.774879    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9" event={"ID":"6b93002e-a2d9-4bd4-9bb3-266576873ebb","Type":"ContainerStarted","Data":"d85d61977c1fb1e32d59b168b387f48a0e0a12119966fe9f40e7e4cc43966cc4"}
Oct 29 22:59:41.774974 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:41.774904    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"
Oct 29 22:59:41.819390 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:41.819277    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k" podStartSLOduration=1.8192576649999999 podStartE2EDuration="1.819257665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:41.813947289 +0000 UTC m=+2720.725390857" watchObservedRunningTime="2026-10-29 22:59:41.819257665 +0000 UTC m=+2720.730701220"
Oct 29 22:59:41.867113 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:41.867056    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9" podStartSLOduration=1.8670410309999999 podStartE2EDuration="1.867041031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:41.865689828 +0000 UTC m=+2720.777133407" watchObservedRunningTime="2026-10-29 22:59:41.867041031 +0000 UTC m=+2720.778484597"
Oct 29 22:59:43.726487 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:59:43.726451    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 22:59:43.770309 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:43.770236    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m" podUID="d838fae2-87e9-4874-bb79-92a34e503ce6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:59:49.587156 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:49.587126    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"
Oct 29 22:59:49.683241 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:49.683203    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d838fae2-87e9-4874-bb79-92a34e503ce6-kube-api-access-sjdhf\" (UniqueName: \"kubernetes.io/projected/d838fae2-87e9-4874-bb79-92a34e503ce6-kube-api-access-sjdhf\") pod \"d838fae2-87e9-4874-bb79-92a34e503ce6\" (UID: \"d838fae2-87e9-4874-bb79-92a34e503ce6\") "
Oct 29 22:59:49.683447 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:49.683407    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d838fae2-87e9-4874-bb79-92a34e503ce6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d838fae2-87e9-4874-bb79-92a34e503ce6-plugins-config-volume\") pod \"d838fae2-87e9-4874-bb79-92a34e503ce6\" (UID: \"d838fae2-87e9-4874-bb79-92a34e503ce6\") "
Oct 29 22:59:49.684110 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:49.684084    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d838fae2-87e9-4874-bb79-92a34e503ce6-plugins-config-volume" pod "d838fae2-87e9-4874-bb79-92a34e503ce6" (UID: "d838fae2-87e9-4874-bb79-92a34e503ce6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:49.684611 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:49.684585    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d838fae2-87e9-4874-bb79-92a34e503ce6-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:49.688169 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:49.686620    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d838fae2-87e9-4874-bb79-92a34e503ce6-kube-api-access-sjdhf" pod "d838fae2-87e9-4874-bb79-92a34e503ce6" (UID: "d838fae2-87e9-4874-bb79-92a34e503ce6"). InnerVolumeSpecName "kube-api-access-sjdhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:49.787394 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:49.787309    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjdhf\" (UniqueName: \"kubernetes.io/projected/d838fae2-87e9-4874-bb79-92a34e503ce6-kube-api-access-sjdhf\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:49.893571 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:49.893532    4559 generic.go:356] "Generic (PLEG): container finished" podID="d838fae2-87e9-4874-bb79-92a34e503ce6" containerID="f302bc68b25b097a604a22004fd9d01e628990029018c4e357687848d577b27c" exitCode=0
Oct 29 22:59:49.893755 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:49.893581    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"
Oct 29 22:59:49.893755 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:49.893611    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m" event={"ID":"d838fae2-87e9-4874-bb79-92a34e503ce6","Type":"ContainerDied","Data":"f302bc68b25b097a604a22004fd9d01e628990029018c4e357687848d577b27c"}
Oct 29 22:59:49.893755 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:49.893672    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m" event={"ID":"d838fae2-87e9-4874-bb79-92a34e503ce6","Type":"ContainerDied","Data":"c47799a4267e0849df316dded57508355f17b439c6e036dbd832173bcc770624"}
Oct 29 22:59:49.893755 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:49.893700    4559 scope.go:122] "RemoveContainer" containerID="f302bc68b25b097a604a22004fd9d01e628990029018c4e357687848d577b27c"
Oct 29 22:59:49.987510 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:49.987485    4559 scope.go:122] "RemoveContainer" containerID="f302bc68b25b097a604a22004fd9d01e628990029018c4e357687848d577b27c"
Oct 29 22:59:49.987949 ip-10-0-138-2 kubenswrapper[4559]: E1029 22:59:49.987921    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f302bc68b25b097a604a22004fd9d01e628990029018c4e357687848d577b27c\": container with ID starting with f302bc68b25b097a604a22004fd9d01e628990029018c4e357687848d577b27c not found: ID does not exist" containerID="f302bc68b25b097a604a22004fd9d01e628990029018c4e357687848d577b27c"
Oct 29 22:59:49.988042 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:49.987957    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f302bc68b25b097a604a22004fd9d01e628990029018c4e357687848d577b27c"} err="failed to get container status \"f302bc68b25b097a604a22004fd9d01e628990029018c4e357687848d577b27c\": rpc error: code = NotFound desc = could not find container \"f302bc68b25b097a604a22004fd9d01e628990029018c4e357687848d577b27c\": container with ID starting with f302bc68b25b097a604a22004fd9d01e628990029018c4e357687848d577b27c not found: ID does not exist"
Oct 29 22:59:49.996045 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:49.996006    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"]
Oct 29 22:59:50.000873 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:50.000837    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa3c6-64445b56bc-9wn9m"]
Oct 29 22:59:50.497746 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:50.495022    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": read tcp 10.134.0.2:48166->10.134.0.230:15021: read: connection reset by peer" start-of-body=
Oct 29 22:59:50.497746 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:50.495091    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r" podUID="ec19a061-f45f-42f2-aea1-da03fe800597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": read tcp 10.134.0.2:48166->10.134.0.230:15021: read: connection reset by peer"
Oct 29 22:59:50.909444 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:50.909405    4559 generic.go:356] "Generic (PLEG): container finished" podID="ec19a061-f45f-42f2-aea1-da03fe800597" containerID="305c5988baf041f6f2d0a220e8df32aec6982d2b2cbd1fce10400f2ed240bebb" exitCode=0
Oct 29 22:59:50.909926 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:50.909474    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r" event={"ID":"ec19a061-f45f-42f2-aea1-da03fe800597","Type":"ContainerDied","Data":"305c5988baf041f6f2d0a220e8df32aec6982d2b2cbd1fce10400f2ed240bebb"}
Oct 29 22:59:50.909977 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:50.909960    4559 scope.go:122] "RemoveContainer" containerID="305c5988baf041f6f2d0a220e8df32aec6982d2b2cbd1fce10400f2ed240bebb"
Oct 29 22:59:50.911942 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:50.911912    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:59:50.912039 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:50.911987    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:59:50.912039 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:50.912018    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 22:59:51.231555 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:51.231454    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d838fae2-87e9-4874-bb79-92a34e503ce6" path="/var/lib/kubelet/pods/d838fae2-87e9-4874-bb79-92a34e503ce6/volumes"
Oct 29 22:59:51.923202 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:51.923165    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r" event={"ID":"ec19a061-f45f-42f2-aea1-da03fe800597","Type":"ContainerStarted","Data":"62c13c07b39673a67462946b5db36cd0aa0df1ff4c43270d4f16df3c34f9a823"}
Oct 29 22:59:52.512226 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:52.512067    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:59:52.512459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:52.512278    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 29 22:59:52.512459 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:52.512327    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r" podUID="ec19a061-f45f-42f2-aea1-da03fe800597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 29 22:59:52.786852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:52.786726    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"
Oct 29 22:59:52.786852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:52.786847    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"
Oct 29 22:59:53.511254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:53.511215    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 29 22:59:53.511742 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:53.511275    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r" podUID="ec19a061-f45f-42f2-aea1-da03fe800597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 29 22:59:54.511852 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:54.511776    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 29 22:59:54.512371 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:54.511872    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r" podUID="ec19a061-f45f-42f2-aea1-da03fe800597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 29 22:59:55.512187 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:55.512142    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 29 22:59:55.512670 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:55.512215    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r" podUID="ec19a061-f45f-42f2-aea1-da03fe800597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 29 22:59:55.831882 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:55.831773    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main" containerID="cri-o://179b810267495e341681e03620615bf5cf1d3cd62b700ab96f85be85de5fe8d6" gracePeriod=45
Oct 29 22:59:56.013456 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.013413    4559 generic.go:356] "Generic (PLEG): container finished" podID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerID="179b810267495e341681e03620615bf5cf1d3cd62b700ab96f85be85de5fe8d6" exitCode=0
Oct 29 22:59:56.013761 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.013585    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" event={"ID":"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f","Type":"ContainerDied","Data":"179b810267495e341681e03620615bf5cf1d3cd62b700ab96f85be85de5fe8d6"}
Oct 29 22:59:56.430465 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.430441    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:59:56.512000 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.511909    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 29 22:59:56.512000 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.511980    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r" podUID="ec19a061-f45f-42f2-aea1-da03fe800597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 29 22:59:56.610617 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.610584    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-home\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-home\") pod \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") "
Oct 29 22:59:56.611062 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.610657    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-dshm\") pod \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") "
Oct 29 22:59:56.611062 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.610683    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-tls-certs\" (UniqueName: \"kubernetes.io/secret/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-tls-certs\") pod \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") "
Oct 29 22:59:56.611062 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.610771    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-kube-api-access-b75zm\" (UniqueName: \"kubernetes.io/projected/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-kube-api-access-b75zm\") pod \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") "
Oct 29 22:59:56.611062 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.610990    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-model-cache\") pod \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") "
Oct 29 22:59:56.611233 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.611081    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-home" pod "96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" (UID: "96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:56.611233 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.611184    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-tmp-dir\") pod \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\" (UID: \"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f\") "
Oct 29 22:59:56.611494 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.611452    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-model-cache" pod "96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" (UID: "96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:56.611898 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.611871    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-tmp-dir" pod "96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" (UID: "96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:56.612056 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.612037    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:56.612117 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.612059    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:56.612117 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.612068    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:56.613984 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.613961    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-tls-certs" pod "96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" (UID: "96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:56.613984 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.613970    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-dshm" pod "96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" (UID: "96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:56.614536 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.614516    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-kube-api-access-b75zm" pod "96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" (UID: "96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f"). InnerVolumeSpecName "kube-api-access-b75zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:56.713921 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.713880    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:56.713921 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.713915    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:56.713921 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:56.713928    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b75zm\" (UniqueName: \"kubernetes.io/projected/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f-kube-api-access-b75zm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 22:59:57.024610 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:57.024582    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"
Oct 29 22:59:57.024848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:57.024620    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f" event={"ID":"96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f","Type":"ContainerDied","Data":"47e08b23930ff5378083e650b8b3c3010ecf4190458275d3326b80d97b29f957"}
Oct 29 22:59:57.024848 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:57.024680    4559 scope.go:122] "RemoveContainer" containerID="179b810267495e341681e03620615bf5cf1d3cd62b700ab96f85be85de5fe8d6"
Oct 29 22:59:57.101100 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:57.101069    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"]
Oct 29 22:59:57.113785 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:57.113746    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/premium-w6-784e7a-kserve-5dd7cf45b4-vqp9f"]
Oct 29 22:59:57.268236 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:57.268197    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" path="/var/lib/kubelet/pods/96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f/volumes"
Oct 29 22:59:57.511523 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:57.511464    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 29 22:59:57.511763 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:57.511545    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r" podUID="ec19a061-f45f-42f2-aea1-da03fe800597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 29 22:59:58.516929 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:58.516893    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:59:58.517438 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:58.517415    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 22:59:58.518254 ip-10-0-138-2 kubenswrapper[4559]: I1029 22:59:58.518220    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 23:00:00.151292 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.151257    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8"]
Oct 29 23:00:00.152256 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.152235    4559 cpu_manager.go:414] "containerMap: removing container" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main"
Oct 29 23:00:00.152341 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.152258    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main"
Oct 29 23:00:00.152341 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.152283    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d838fae2-87e9-4874-bb79-92a34e503ce6" containerName="payload-pre-processing"
Oct 29 23:00:00.152341 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.152289    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d838fae2-87e9-4874-bb79-92a34e503ce6" containerName="payload-pre-processing"
Oct 29 23:00:00.152443 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.152404    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d9ca2d-e01b-4bf4-8cfc-2d0fe145e51f" containerName="main"
Oct 29 23:00:00.152443 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.152416    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d838fae2-87e9-4874-bb79-92a34e503ce6" containerName="payload-pre-processing"
Oct 29 23:00:00.157842 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.157792    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg"]
Oct 29 23:00:00.158006 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.157933    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8"
Oct 29 23:00:00.163165 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.162794    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg"
Oct 29 23:00:00.170926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.170641    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ae42cd-1e69-44b6-b728-fb34e15e46c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8\" (UID: \"64ae42cd-1e69-44b6-b728-fb34e15e46c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8"
Oct 29 23:00:00.171062 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.171029    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqdph\" (UniqueName: \"kubernetes.io/projected/64ae42cd-1e69-44b6-b728-fb34e15e46c7-kube-api-access-fqdph\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8\" (UID: \"64ae42cd-1e69-44b6-b728-fb34e15e46c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8"
Oct 29 23:00:00.176062 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.175265    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr"]
Oct 29 23:00:00.182685 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.182653    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw"]
Oct 29 23:00:00.191834 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.187574    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr"
Oct 29 23:00:00.191834 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.189661    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:00:00.213906 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.210972    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8"]
Oct 29 23:00:00.213906 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.211119    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw"
Oct 29 23:00:00.232115 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.221232    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4"]
Oct 29 23:00:00.232115 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.231149    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4"
Oct 29 23:00:00.263393 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.263356    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg"]
Oct 29 23:00:00.269504 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.269469    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr"]
Oct 29 23:00:00.278585 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.274457    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4477\" (UniqueName: \"kubernetes.io/projected/619b371d-35a7-43d9-ace8-cfc478f2fdfa-kube-api-access-j4477\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw\" (UID: \"619b371d-35a7-43d9-ace8-cfc478f2fdfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw"
Oct 29 23:00:00.278585 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.274530    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w9w4\" (UniqueName: \"kubernetes.io/projected/bb6a5323-5dab-421e-9c50-b28825aaa895-kube-api-access-8w9w4\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg\" (UID: \"bb6a5323-5dab-421e-9c50-b28825aaa895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg"
Oct 29 23:00:00.278585 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.274690    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ts8c\" (UniqueName: \"kubernetes.io/projected/624b0508-b13d-41aa-977f-8691e3b0ca01-kube-api-access-9ts8c\") pod \"maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr\" (UID: \"624b0508-b13d-41aa-977f-8691e3b0ca01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr"
Oct 29 23:00:00.278585 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.274772    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q2tn\" (UniqueName: \"kubernetes.io/projected/a2e53e77-e3c5-4468-b830-79e768275dde-kube-api-access-2q2tn\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4\" (UID: \"a2e53e77-e3c5-4468-b830-79e768275dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4"
Oct 29 23:00:00.278585 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.274827    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e53e77-e3c5-4468-b830-79e768275dde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4\" (UID: \"a2e53e77-e3c5-4468-b830-79e768275dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4"
Oct 29 23:00:00.278585 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.274888    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqdph\" (UniqueName: \"kubernetes.io/projected/64ae42cd-1e69-44b6-b728-fb34e15e46c7-kube-api-access-fqdph\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8\" (UID: \"64ae42cd-1e69-44b6-b728-fb34e15e46c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8"
Oct 29 23:00:00.278585 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.275028    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6a5323-5dab-421e-9c50-b28825aaa895-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg\" (UID: \"bb6a5323-5dab-421e-9c50-b28825aaa895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg"
Oct 29 23:00:00.278585 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.275068    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/624b0508-b13d-41aa-977f-8691e3b0ca01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr\" (UID: \"624b0508-b13d-41aa-977f-8691e3b0ca01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr"
Oct 29 23:00:00.278585 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.275123    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ae42cd-1e69-44b6-b728-fb34e15e46c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8\" (UID: \"64ae42cd-1e69-44b6-b728-fb34e15e46c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8"
Oct 29 23:00:00.278585 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.275178    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619b371d-35a7-43d9-ace8-cfc478f2fdfa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw\" (UID: \"619b371d-35a7-43d9-ace8-cfc478f2fdfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw"
Oct 29 23:00:00.279777 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.279746    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw"]
Oct 29 23:00:00.280956 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.280930    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ae42cd-1e69-44b6-b728-fb34e15e46c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8\" (UID: \"64ae42cd-1e69-44b6-b728-fb34e15e46c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8"
Oct 29 23:00:00.294079 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.294042    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd"]
Oct 29 23:00:00.316029 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.315999    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr"]
Oct 29 23:00:00.317108 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.317084    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd"
Oct 29 23:00:00.329044 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.328933    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqdph\" (UniqueName: \"kubernetes.io/projected/64ae42cd-1e69-44b6-b728-fb34e15e46c7-kube-api-access-fqdph\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8\" (UID: \"64ae42cd-1e69-44b6-b728-fb34e15e46c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8"
Oct 29 23:00:00.336564 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.336533    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g"]
Oct 29 23:00:00.336797 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.336765    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr"
Oct 29 23:00:00.343937 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.343904    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd"]
Oct 29 23:00:00.344148 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.343959    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt"]
Oct 29 23:00:00.344602 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.344564    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g"
Oct 29 23:00:00.352125 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.352089    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr"]
Oct 29 23:00:00.352284 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.352136    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4"]
Oct 29 23:00:00.352284 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.352229    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt"
Oct 29 23:00:00.355698 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.355672    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7"]
Oct 29 23:00:00.362211 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.362026    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g"]
Oct 29 23:00:00.362211 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.362148    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7"
Oct 29 23:00:00.370680 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.370651    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4"]
Oct 29 23:00:00.376602 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.376571    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw"]
Oct 29 23:00:00.376958 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.376932    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w9w4\" (UniqueName: \"kubernetes.io/projected/bb6a5323-5dab-421e-9c50-b28825aaa895-kube-api-access-8w9w4\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg\" (UID: \"bb6a5323-5dab-421e-9c50-b28825aaa895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg"
Oct 29 23:00:00.377121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.377102    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml76t\" (UniqueName: \"kubernetes.io/projected/32ea5735-9ca9-4980-8031-44fe6c4269ef-kube-api-access-ml76t\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr\" (UID: \"32ea5735-9ca9-4980-8031-44fe6c4269ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr"
Oct 29 23:00:00.377275 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.377260    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ea5735-9ca9-4980-8031-44fe6c4269ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr\" (UID: \"32ea5735-9ca9-4980-8031-44fe6c4269ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr"
Oct 29 23:00:00.377396 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.377382    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbspv\" (UniqueName: \"kubernetes.io/projected/58b25906-470a-4dbd-b43a-0f0bcd25223c-kube-api-access-mbspv\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7\" (UID: \"58b25906-470a-4dbd-b43a-0f0bcd25223c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7"
Oct 29 23:00:00.377526 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.377510    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ts8c\" (UniqueName: \"kubernetes.io/projected/624b0508-b13d-41aa-977f-8691e3b0ca01-kube-api-access-9ts8c\") pod \"maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr\" (UID: \"624b0508-b13d-41aa-977f-8691e3b0ca01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr"
Oct 29 23:00:00.377651 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.377637    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2tn\" (UniqueName: \"kubernetes.io/projected/a2e53e77-e3c5-4468-b830-79e768275dde-kube-api-access-2q2tn\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4\" (UID: \"a2e53e77-e3c5-4468-b830-79e768275dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4"
Oct 29 23:00:00.377759 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.377745    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e53e77-e3c5-4468-b830-79e768275dde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4\" (UID: \"a2e53e77-e3c5-4468-b830-79e768275dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4"
Oct 29 23:00:00.377972 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.377954    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wfb5\" (UniqueName: \"kubernetes.io/projected/f6fdff4a-3510-4774-8c07-dddad4aa7219-kube-api-access-2wfb5\") pod \"maas-api-key-cleanup-29888580-bpvzt\" (UID: \"f6fdff4a-3510-4774-8c07-dddad4aa7219\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt"
Oct 29 23:00:00.378107 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.378092    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6a5323-5dab-421e-9c50-b28825aaa895-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg\" (UID: \"bb6a5323-5dab-421e-9c50-b28825aaa895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg"
Oct 29 23:00:00.378239 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.378224    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/624b0508-b13d-41aa-977f-8691e3b0ca01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr\" (UID: \"624b0508-b13d-41aa-977f-8691e3b0ca01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr"
Oct 29 23:00:00.378376 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.378363    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619b371d-35a7-43d9-ace8-cfc478f2fdfa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw\" (UID: \"619b371d-35a7-43d9-ace8-cfc478f2fdfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw"
Oct 29 23:00:00.378489 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.378474    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8g2d\" (UniqueName: \"kubernetes.io/projected/3b1c42ea-be0a-4b54-b41c-cd7116493b54-kube-api-access-g8g2d\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g\" (UID: \"3b1c42ea-be0a-4b54-b41c-cd7116493b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g"
Oct 29 23:00:00.378663 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.378646    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b25906-470a-4dbd-b43a-0f0bcd25223c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7\" (UID: \"58b25906-470a-4dbd-b43a-0f0bcd25223c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7"
Oct 29 23:00:00.378789 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.378766    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2dcm\" (UniqueName: \"kubernetes.io/projected/7498470d-5bab-4716-8feb-c0e70788cc88-kube-api-access-c2dcm\") pod \"maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd\" (UID: \"7498470d-5bab-4716-8feb-c0e70788cc88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd"
Oct 29 23:00:00.378940 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.378858    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c42ea-be0a-4b54-b41c-cd7116493b54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g\" (UID: \"3b1c42ea-be0a-4b54-b41c-cd7116493b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g"
Oct 29 23:00:00.378940 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.378894    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7498470d-5bab-4716-8feb-c0e70788cc88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd\" (UID: \"7498470d-5bab-4716-8feb-c0e70788cc88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd"
Oct 29 23:00:00.378940 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.378920    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6fdff4a-3510-4774-8c07-dddad4aa7219-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888580-bpvzt\" (UID: \"f6fdff4a-3510-4774-8c07-dddad4aa7219\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt"
Oct 29 23:00:00.379095 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.379077    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4477\" (UniqueName: \"kubernetes.io/projected/619b371d-35a7-43d9-ace8-cfc478f2fdfa-kube-api-access-j4477\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw\" (UID: \"619b371d-35a7-43d9-ace8-cfc478f2fdfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw"
Oct 29 23:00:00.380785 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.380761    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e53e77-e3c5-4468-b830-79e768275dde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4\" (UID: \"a2e53e77-e3c5-4468-b830-79e768275dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4"
Oct 29 23:00:00.381488 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.381463    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6a5323-5dab-421e-9c50-b28825aaa895-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg\" (UID: \"bb6a5323-5dab-421e-9c50-b28825aaa895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg"
Oct 29 23:00:00.381871 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.381778    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/624b0508-b13d-41aa-977f-8691e3b0ca01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr\" (UID: \"624b0508-b13d-41aa-977f-8691e3b0ca01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr"
Oct 29 23:00:00.381871 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.381778    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619b371d-35a7-43d9-ace8-cfc478f2fdfa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw\" (UID: \"619b371d-35a7-43d9-ace8-cfc478f2fdfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw"
Oct 29 23:00:00.382036 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.381880    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4"
Oct 29 23:00:00.384388 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.384366    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt"]
Oct 29 23:00:00.384509 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.384465    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw"
Oct 29 23:00:00.389064 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.389038    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7"]
Oct 29 23:00:00.400125 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.400016    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4"]
Oct 29 23:00:00.400394 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.400312    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w9w4\" (UniqueName: \"kubernetes.io/projected/bb6a5323-5dab-421e-9c50-b28825aaa895-kube-api-access-8w9w4\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg\" (UID: \"bb6a5323-5dab-421e-9c50-b28825aaa895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg"
Oct 29 23:00:00.400551 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.400525    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q2tn\" (UniqueName: \"kubernetes.io/projected/a2e53e77-e3c5-4468-b830-79e768275dde-kube-api-access-2q2tn\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4\" (UID: \"a2e53e77-e3c5-4468-b830-79e768275dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4"
Oct 29 23:00:00.400617 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.400534    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ts8c\" (UniqueName: \"kubernetes.io/projected/624b0508-b13d-41aa-977f-8691e3b0ca01-kube-api-access-9ts8c\") pod \"maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr\" (UID: \"624b0508-b13d-41aa-977f-8691e3b0ca01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr"
Oct 29 23:00:00.401244 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.401221    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4477\" (UniqueName: \"kubernetes.io/projected/619b371d-35a7-43d9-ace8-cfc478f2fdfa-kube-api-access-j4477\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw\" (UID: \"619b371d-35a7-43d9-ace8-cfc478f2fdfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw"
Oct 29 23:00:00.410394 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.410216    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw"]
Oct 29 23:00:00.480651 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.480594    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb88948-9bd8-4690-bb16-0e4e5cebb1e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw\" (UID: \"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw"
Oct 29 23:00:00.480867 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.480765    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml76t\" (UniqueName: \"kubernetes.io/projected/32ea5735-9ca9-4980-8031-44fe6c4269ef-kube-api-access-ml76t\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr\" (UID: \"32ea5735-9ca9-4980-8031-44fe6c4269ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr"
Oct 29 23:00:00.480867 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.480859    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ea5735-9ca9-4980-8031-44fe6c4269ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr\" (UID: \"32ea5735-9ca9-4980-8031-44fe6c4269ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr"
Oct 29 23:00:00.480983 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.480882    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv6pq\" (UniqueName: \"kubernetes.io/projected/40c6ad42-57b2-41c2-a6a5-30fa68a808d4-kube-api-access-tv6pq\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4\" (UID: \"40c6ad42-57b2-41c2-a6a5-30fa68a808d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4"
Oct 29 23:00:00.480983 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.480909    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbspv\" (UniqueName: \"kubernetes.io/projected/58b25906-470a-4dbd-b43a-0f0bcd25223c-kube-api-access-mbspv\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7\" (UID: \"58b25906-470a-4dbd-b43a-0f0bcd25223c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7"
Oct 29 23:00:00.481070 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.481020    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5kzx\" (UniqueName: \"kubernetes.io/projected/0fb88948-9bd8-4690-bb16-0e4e5cebb1e3-kube-api-access-w5kzx\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw\" (UID: \"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw"
Oct 29 23:00:00.481145 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.481132    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wfb5\" (UniqueName: \"kubernetes.io/projected/f6fdff4a-3510-4774-8c07-dddad4aa7219-kube-api-access-2wfb5\") pod \"maas-api-key-cleanup-29888580-bpvzt\" (UID: \"f6fdff4a-3510-4774-8c07-dddad4aa7219\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt"
Oct 29 23:00:00.481213 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.481201    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c6ad42-57b2-41c2-a6a5-30fa68a808d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4\" (UID: \"40c6ad42-57b2-41c2-a6a5-30fa68a808d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4"
Oct 29 23:00:00.481320 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.481299    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8g2d\" (UniqueName: \"kubernetes.io/projected/3b1c42ea-be0a-4b54-b41c-cd7116493b54-kube-api-access-g8g2d\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g\" (UID: \"3b1c42ea-be0a-4b54-b41c-cd7116493b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g"
Oct 29 23:00:00.481422 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.481405    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b25906-470a-4dbd-b43a-0f0bcd25223c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7\" (UID: \"58b25906-470a-4dbd-b43a-0f0bcd25223c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7"
Oct 29 23:00:00.481484 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.481437    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2dcm\" (UniqueName: \"kubernetes.io/projected/7498470d-5bab-4716-8feb-c0e70788cc88-kube-api-access-c2dcm\") pod \"maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd\" (UID: \"7498470d-5bab-4716-8feb-c0e70788cc88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd"
Oct 29 23:00:00.481543 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.481495    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c42ea-be0a-4b54-b41c-cd7116493b54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g\" (UID: \"3b1c42ea-be0a-4b54-b41c-cd7116493b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g"
Oct 29 23:00:00.481543 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.481531    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7498470d-5bab-4716-8feb-c0e70788cc88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd\" (UID: \"7498470d-5bab-4716-8feb-c0e70788cc88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd"
Oct 29 23:00:00.481648 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.481557    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6fdff4a-3510-4774-8c07-dddad4aa7219-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888580-bpvzt\" (UID: \"f6fdff4a-3510-4774-8c07-dddad4aa7219\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt"
Oct 29 23:00:00.481966 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.481943    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ea5735-9ca9-4980-8031-44fe6c4269ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr\" (UID: \"32ea5735-9ca9-4980-8031-44fe6c4269ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr"
Oct 29 23:00:00.482228 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.482202    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b25906-470a-4dbd-b43a-0f0bcd25223c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7\" (UID: \"58b25906-470a-4dbd-b43a-0f0bcd25223c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7"
Oct 29 23:00:00.482280 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.482252    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c42ea-be0a-4b54-b41c-cd7116493b54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g\" (UID: \"3b1c42ea-be0a-4b54-b41c-cd7116493b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g"
Oct 29 23:00:00.482340 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.482318    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7498470d-5bab-4716-8feb-c0e70788cc88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd\" (UID: \"7498470d-5bab-4716-8feb-c0e70788cc88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd"
Oct 29 23:00:00.482398 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.482382    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6fdff4a-3510-4774-8c07-dddad4aa7219-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888580-bpvzt\" (UID: \"f6fdff4a-3510-4774-8c07-dddad4aa7219\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt"
Oct 29 23:00:00.488853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.488832    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8"
Oct 29 23:00:00.502181 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.502156    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wfb5\" (UniqueName: \"kubernetes.io/projected/f6fdff4a-3510-4774-8c07-dddad4aa7219-kube-api-access-2wfb5\") pod \"maas-api-key-cleanup-29888580-bpvzt\" (UID: \"f6fdff4a-3510-4774-8c07-dddad4aa7219\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt"
Oct 29 23:00:00.502381 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.502201    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbspv\" (UniqueName: \"kubernetes.io/projected/58b25906-470a-4dbd-b43a-0f0bcd25223c-kube-api-access-mbspv\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7\" (UID: \"58b25906-470a-4dbd-b43a-0f0bcd25223c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7"
Oct 29 23:00:00.502478 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.502460    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2dcm\" (UniqueName: \"kubernetes.io/projected/7498470d-5bab-4716-8feb-c0e70788cc88-kube-api-access-c2dcm\") pod \"maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd\" (UID: \"7498470d-5bab-4716-8feb-c0e70788cc88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd"
Oct 29 23:00:00.503016 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.502979    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8g2d\" (UniqueName: \"kubernetes.io/projected/3b1c42ea-be0a-4b54-b41c-cd7116493b54-kube-api-access-g8g2d\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g\" (UID: \"3b1c42ea-be0a-4b54-b41c-cd7116493b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g"
Oct 29 23:00:00.503336 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.503314    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml76t\" (UniqueName: \"kubernetes.io/projected/32ea5735-9ca9-4980-8031-44fe6c4269ef-kube-api-access-ml76t\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr\" (UID: \"32ea5735-9ca9-4980-8031-44fe6c4269ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr"
Oct 29 23:00:00.559976 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.559942    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg"
Oct 29 23:00:00.605267 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.595462    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv6pq\" (UniqueName: \"kubernetes.io/projected/40c6ad42-57b2-41c2-a6a5-30fa68a808d4-kube-api-access-tv6pq\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4\" (UID: \"40c6ad42-57b2-41c2-a6a5-30fa68a808d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4"
Oct 29 23:00:00.605267 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.595574    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5kzx\" (UniqueName: \"kubernetes.io/projected/0fb88948-9bd8-4690-bb16-0e4e5cebb1e3-kube-api-access-w5kzx\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw\" (UID: \"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw"
Oct 29 23:00:00.605267 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.595670    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c6ad42-57b2-41c2-a6a5-30fa68a808d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4\" (UID: \"40c6ad42-57b2-41c2-a6a5-30fa68a808d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4"
Oct 29 23:00:00.605267 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.595768    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb88948-9bd8-4690-bb16-0e4e5cebb1e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw\" (UID: \"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw"
Oct 29 23:00:00.605267 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.597645    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb88948-9bd8-4690-bb16-0e4e5cebb1e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw\" (UID: \"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw"
Oct 29 23:00:00.605267 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.599832    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c6ad42-57b2-41c2-a6a5-30fa68a808d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4\" (UID: \"40c6ad42-57b2-41c2-a6a5-30fa68a808d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4"
Oct 29 23:00:00.605267 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.600075    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr"
Oct 29 23:00:00.621359 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.621168    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv6pq\" (UniqueName: \"kubernetes.io/projected/40c6ad42-57b2-41c2-a6a5-30fa68a808d4-kube-api-access-tv6pq\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4\" (UID: \"40c6ad42-57b2-41c2-a6a5-30fa68a808d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4"
Oct 29 23:00:00.624706 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.624672    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5kzx\" (UniqueName: \"kubernetes.io/projected/0fb88948-9bd8-4690-bb16-0e4e5cebb1e3-kube-api-access-w5kzx\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw\" (UID: \"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw"
Oct 29 23:00:00.632798 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.632761    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw"
Oct 29 23:00:00.636884 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.636838    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4"
Oct 29 23:00:00.673131 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.672875    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd"
Oct 29 23:00:00.686993 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.686914    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr"
Oct 29 23:00:00.700013 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.699924    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g"
Oct 29 23:00:00.730756 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.730707    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt"
Oct 29 23:00:00.742484 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.741830    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7"
Oct 29 23:00:00.754997 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.753664    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4"
Oct 29 23:00:00.762431 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.762185    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw"
Oct 29 23:00:00.846828 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:00.846768    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8"]
Oct 29 23:00:00.890761 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:00:00.890718    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ae42cd_1e69_44b6_b728_fb34e15e46c7.slice/crio-596ede21fab01675ac5f73cb3b8407c54f1fc5d55c3c681af765e29ab8c0f25e WatchSource:0}: Error finding container 596ede21fab01675ac5f73cb3b8407c54f1fc5d55c3c681af765e29ab8c0f25e: Status 404 returned error can't find the container with id 596ede21fab01675ac5f73cb3b8407c54f1fc5d55c3c681af765e29ab8c0f25e
Oct 29 23:00:01.112911 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:01.112866    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8" event={"ID":"64ae42cd-1e69-44b6-b728-fb34e15e46c7","Type":"ContainerStarted","Data":"596ede21fab01675ac5f73cb3b8407c54f1fc5d55c3c681af765e29ab8c0f25e"}
Oct 29 23:00:01.149463 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:01.149428    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg"]
Oct 29 23:00:01.236978 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:00:01.236900    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6a5323_5dab_421e_9c50_b28825aaa895.slice/crio-d7194d6cf5ba1f579b9795cf8f4034500ef4ff38c73212453c956d55565cbc82 WatchSource:0}: Error finding container d7194d6cf5ba1f579b9795cf8f4034500ef4ff38c73212453c956d55565cbc82: Status 404 returned error can't find the container with id d7194d6cf5ba1f579b9795cf8f4034500ef4ff38c73212453c956d55565cbc82
Oct 29 23:00:01.350372 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:01.350336    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw"]
Oct 29 23:00:01.351965 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:00:01.351929    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619b371d_35a7_43d9_ace8_cfc478f2fdfa.slice/crio-a01c4fcd89df91dee1eacc7e2c1842f675c87110d1ad6e7e76a20175200b3d19 WatchSource:0}: Error finding container a01c4fcd89df91dee1eacc7e2c1842f675c87110d1ad6e7e76a20175200b3d19: Status 404 returned error can't find the container with id a01c4fcd89df91dee1eacc7e2c1842f675c87110d1ad6e7e76a20175200b3d19
Oct 29 23:00:01.375225 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:01.375189    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr"]
Oct 29 23:00:01.637074 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:01.636973    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4"]
Oct 29 23:00:01.680004 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:01.679004    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7"]
Oct 29 23:00:01.721975 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:00:01.721908    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-f43c4188e8c5830f8c57b6f84983c8f5e23963357644e3a66142bcca88b4ae31 WatchSource:0}: Error finding container f43c4188e8c5830f8c57b6f84983c8f5e23963357644e3a66142bcca88b4ae31: Status 404 returned error can't find the container with id f43c4188e8c5830f8c57b6f84983c8f5e23963357644e3a66142bcca88b4ae31
Oct 29 23:00:02.076046 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:01.884462    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd"]
Oct 29 23:00:02.171263 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:02.171220    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g"]
Oct 29 23:00:02.185915 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:02.185882    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt"]
Oct 29 23:00:02.189170 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:02.189128    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4" event={"ID":"a2e53e77-e3c5-4468-b830-79e768275dde","Type":"ContainerStarted","Data":"f43c4188e8c5830f8c57b6f84983c8f5e23963357644e3a66142bcca88b4ae31"}
Oct 29 23:00:02.200406 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:02.200331    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7" event={"ID":"58b25906-470a-4dbd-b43a-0f0bcd25223c","Type":"ContainerStarted","Data":"bca8dacb55ac83d208bad8cf12714844ca3b6a8400ec5969a199ff3f926711e2"}
Oct 29 23:00:02.215062 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:02.214935    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr"]
Oct 29 23:00:02.215606 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:02.215576    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr" event={"ID":"32ea5735-9ca9-4980-8031-44fe6c4269ef","Type":"ContainerStarted","Data":"3a6b4fc5b7bfe89b6918e4eb70886bef1b50defd50395213527f8581405ce994"}
Oct 29 23:00:02.241650 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:02.224134    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw"]
Oct 29 23:00:02.245132 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:02.244388    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4"]
Oct 29 23:00:02.247604 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:02.247295    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw" event={"ID":"619b371d-35a7-43d9-ace8-cfc478f2fdfa","Type":"ContainerStarted","Data":"a01c4fcd89df91dee1eacc7e2c1842f675c87110d1ad6e7e76a20175200b3d19"}
Oct 29 23:00:02.266102 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:02.265995    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd" event={"ID":"7498470d-5bab-4716-8feb-c0e70788cc88","Type":"ContainerStarted","Data":"5053bd63cf8d7f0097775ddd3eda477096fcf52b8ccc4ad90d23e56042336f7d"}
Oct 29 23:00:02.267525 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:00:02.267479    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c6ad42_57b2_41c2_a6a5_30fa68a808d4.slice/crio-4e2005014fbc9437f04000901cc05e32956d84c93f9293ea0d678eab7594a52a WatchSource:0}: Error finding container 4e2005014fbc9437f04000901cc05e32956d84c93f9293ea0d678eab7594a52a: Status 404 returned error can't find the container with id 4e2005014fbc9437f04000901cc05e32956d84c93f9293ea0d678eab7594a52a
Oct 29 23:00:02.271666 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:02.271612    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg" event={"ID":"bb6a5323-5dab-421e-9c50-b28825aaa895","Type":"ContainerStarted","Data":"d7194d6cf5ba1f579b9795cf8f4034500ef4ff38c73212453c956d55565cbc82"}
Oct 29 23:00:02.276192 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:02.275370    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr" event={"ID":"624b0508-b13d-41aa-977f-8691e3b0ca01","Type":"ContainerStarted","Data":"f262a07aebde4a9778832616738d7eb0dc20b6661978343bfc7e569a82c5c003"}
Oct 29 23:00:02.325659 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:02.325595    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg" podStartSLOduration=2.325575151 podStartE2EDuration="2.325575151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.31435114 +0000 UTC m=+2741.225794691" watchObservedRunningTime="2026-10-29 23:00:02.325575151 +0000 UTC m=+2741.237018718"
Oct 29 23:00:02.361389 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:02.361332    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr" podStartSLOduration=2.361314125 podStartE2EDuration="2.361314125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.358570638 +0000 UTC m=+2741.270014194" watchObservedRunningTime="2026-10-29 23:00:02.361314125 +0000 UTC m=+2741.272757683"
Oct 29 23:00:03.287354 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.287255    4559 generic.go:356] "Generic (PLEG): container finished" podID="619b371d-35a7-43d9-ace8-cfc478f2fdfa" containerID="2569d5e7307b5b4b56240adc5de9dadb54adfd74bb4d0f2eac9de75d162d5124" exitCode=60
Oct 29 23:00:03.287872 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.287413    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw" event={"ID":"619b371d-35a7-43d9-ace8-cfc478f2fdfa","Type":"ContainerDied","Data":"2569d5e7307b5b4b56240adc5de9dadb54adfd74bb4d0f2eac9de75d162d5124"}
Oct 29 23:00:03.287872 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.287777    4559 scope.go:122] "RemoveContainer" containerID="2569d5e7307b5b4b56240adc5de9dadb54adfd74bb4d0f2eac9de75d162d5124"
Oct 29 23:00:03.289375 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.289353    4559 generic.go:356] "Generic (PLEG): container finished" podID="64ae42cd-1e69-44b6-b728-fb34e15e46c7" containerID="4a410020d1d395dd309d754613916224bdef4abb18f211009c0d426e81026bba" exitCode=60
Oct 29 23:00:03.289482 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.289429    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8" event={"ID":"64ae42cd-1e69-44b6-b728-fb34e15e46c7","Type":"ContainerDied","Data":"4a410020d1d395dd309d754613916224bdef4abb18f211009c0d426e81026bba"}
Oct 29 23:00:03.289742 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.289723    4559 scope.go:122] "RemoveContainer" containerID="4a410020d1d395dd309d754613916224bdef4abb18f211009c0d426e81026bba"
Oct 29 23:00:03.291753 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.291727    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4" event={"ID":"40c6ad42-57b2-41c2-a6a5-30fa68a808d4","Type":"ContainerStarted","Data":"7bf0c1e24d8fdcade31a317043042ef1d87e099ba88f89b44f6bc0bad5150941"}
Oct 29 23:00:03.291884 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.291779    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4" event={"ID":"40c6ad42-57b2-41c2-a6a5-30fa68a808d4","Type":"ContainerStarted","Data":"4e2005014fbc9437f04000901cc05e32956d84c93f9293ea0d678eab7594a52a"}
Oct 29 23:00:03.293583 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.293561    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt" event={"ID":"f6fdff4a-3510-4774-8c07-dddad4aa7219","Type":"ContainerStarted","Data":"eb53db022d8df0b2a92c819b7b75f2cb58fd04ea834e1eab144a9b3f9de10f81"}
Oct 29 23:00:03.293661 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.293611    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt" event={"ID":"f6fdff4a-3510-4774-8c07-dddad4aa7219","Type":"ContainerStarted","Data":"f1f9690820c6252d366d6f783b00df51a9db45497476b6c42cc45922fa3bdcdf"}
Oct 29 23:00:03.295392 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.295372    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g" event={"ID":"3b1c42ea-be0a-4b54-b41c-cd7116493b54","Type":"ContainerStarted","Data":"7f4d6c5904a39a24ce521328393ef25b9b5c7e9a7dd4cc0beafa7d341d0fa829"}
Oct 29 23:00:03.295474 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.295410    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g" event={"ID":"3b1c42ea-be0a-4b54-b41c-cd7116493b54","Type":"ContainerStarted","Data":"f71bbbc4f93a3b901e8ae8ecc5068c0b76ebdf28494c16d82560b74c05ea12b2"}
Oct 29 23:00:03.296878 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.296856    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd" event={"ID":"7498470d-5bab-4716-8feb-c0e70788cc88","Type":"ContainerStarted","Data":"041ee35467d0f895bb50a242fb2ab3a38e29f7fc2bf259739e0de2206cf80a17"}
Oct 29 23:00:03.298470 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.298446    4559 generic.go:356] "Generic (PLEG): container finished" podID="bb6a5323-5dab-421e-9c50-b28825aaa895" containerID="66762401ead96d9624811aef403ede11b3ea2efa9fb08e7cb0bc5d86a0fe40a0" exitCode=60
Oct 29 23:00:03.298591 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.298526    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg" event={"ID":"bb6a5323-5dab-421e-9c50-b28825aaa895","Type":"ContainerDied","Data":"66762401ead96d9624811aef403ede11b3ea2efa9fb08e7cb0bc5d86a0fe40a0"}
Oct 29 23:00:03.298820 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.298778    4559 scope.go:122] "RemoveContainer" containerID="66762401ead96d9624811aef403ede11b3ea2efa9fb08e7cb0bc5d86a0fe40a0"
Oct 29 23:00:03.300614 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.300589    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw" event={"ID":"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3","Type":"ContainerStarted","Data":"211e21d4c283e4441fb7d6d02699d10166bcb6c874096c9698a152611f0fd716"}
Oct 29 23:00:03.300724 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.300670    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw" event={"ID":"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3","Type":"ContainerStarted","Data":"c6da5f48993cd16dd8607bcf937f6409ae9c865e7a67bdbe8c4c5caa521e60d1"}
Oct 29 23:00:03.302469 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.302448    4559 generic.go:356] "Generic (PLEG): container finished" podID="624b0508-b13d-41aa-977f-8691e3b0ca01" containerID="9219d6b9e5593ab08156b574e8445b4733bf30738834aabc61153450f8418f1c" exitCode=60
Oct 29 23:00:03.302571 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.302553    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr" event={"ID":"624b0508-b13d-41aa-977f-8691e3b0ca01","Type":"ContainerDied","Data":"9219d6b9e5593ab08156b574e8445b4733bf30738834aabc61153450f8418f1c"}
Oct 29 23:00:03.302856 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.302840    4559 scope.go:122] "RemoveContainer" containerID="9219d6b9e5593ab08156b574e8445b4733bf30738834aabc61153450f8418f1c"
Oct 29 23:00:03.304437 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.304415    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4" event={"ID":"a2e53e77-e3c5-4468-b830-79e768275dde","Type":"ContainerStarted","Data":"ec1338da3b00de30b953c5dc321ee3d8087d4771bb90d4c1db304cc521b7eafb"}
Oct 29 23:00:03.306226 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.306204    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7" event={"ID":"58b25906-470a-4dbd-b43a-0f0bcd25223c","Type":"ContainerStarted","Data":"0a72289948f608497898049679c5781981ba7c01aa4a82a7a32f3ee113e8754c"}
Oct 29 23:00:03.312078 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.312047    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr" event={"ID":"32ea5735-9ca9-4980-8031-44fe6c4269ef","Type":"ContainerStarted","Data":"fbe07017ed7b1273b8189a077e69c2dd1edf6069064654c47e85209dc8d4130e"}
Oct 29 23:00:03.373097 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.373035    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4" podStartSLOduration=3.373019316 podStartE2EDuration="3.373019316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.365986437 +0000 UTC m=+2742.277429990" watchObservedRunningTime="2026-10-29 23:00:03.373019316 +0000 UTC m=+2742.284462916"
Oct 29 23:00:03.405159 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.405110    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr" podStartSLOduration=3.405092167 podStartE2EDuration="3.405092167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.403879098 +0000 UTC m=+2742.315322652" watchObservedRunningTime="2026-10-29 23:00:03.405092167 +0000 UTC m=+2742.316535725"
Oct 29 23:00:03.444660 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.444600    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt" podStartSLOduration=3.4445840309999998 podStartE2EDuration="3.444584031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.441721798 +0000 UTC m=+2742.353165353" watchObservedRunningTime="2026-10-29 23:00:03.444584031 +0000 UTC m=+2742.356027584"
Oct 29 23:00:03.517355 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.517300    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw" podStartSLOduration=3.5172849619999997 podStartE2EDuration="3.517284962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.516625099 +0000 UTC m=+2742.428068651" watchObservedRunningTime="2026-10-29 23:00:03.517284962 +0000 UTC m=+2742.428728516"
Oct 29 23:00:03.561394 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.561331    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd" podStartSLOduration=3.56131515 podStartE2EDuration="3.56131515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.555212583 +0000 UTC m=+2742.466656138" watchObservedRunningTime="2026-10-29 23:00:03.56131515 +0000 UTC m=+2742.472758704"
Oct 29 23:00:03.720093 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.720028    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7" podStartSLOduration=3.720007934 podStartE2EDuration="3.720007934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.637056161 +0000 UTC m=+2742.548499718" watchObservedRunningTime="2026-10-29 23:00:03.720007934 +0000 UTC m=+2742.631451493"
Oct 29 23:00:03.745337 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.745277    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4" podStartSLOduration=3.745257899 podStartE2EDuration="3.745257899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.738762717 +0000 UTC m=+2742.650206272" watchObservedRunningTime="2026-10-29 23:00:03.745257899 +0000 UTC m=+2742.656701456"
Oct 29 23:00:03.777379 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:03.777329    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g" podStartSLOduration=3.7773136430000003 podStartE2EDuration="3.777313643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.776752618 +0000 UTC m=+2742.688196173" watchObservedRunningTime="2026-10-29 23:00:03.777313643 +0000 UTC m=+2742.688757250"
Oct 29 23:00:04.325721 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.325679    4559 generic.go:356] "Generic (PLEG): container finished" podID="3b1c42ea-be0a-4b54-b41c-cd7116493b54" containerID="7f4d6c5904a39a24ce521328393ef25b9b5c7e9a7dd4cc0beafa7d341d0fa829" exitCode=60
Oct 29 23:00:04.326268 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.325723    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g" event={"ID":"3b1c42ea-be0a-4b54-b41c-cd7116493b54","Type":"ContainerDied","Data":"7f4d6c5904a39a24ce521328393ef25b9b5c7e9a7dd4cc0beafa7d341d0fa829"}
Oct 29 23:00:04.326268 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.326198    4559 scope.go:122] "RemoveContainer" containerID="7f4d6c5904a39a24ce521328393ef25b9b5c7e9a7dd4cc0beafa7d341d0fa829"
Oct 29 23:00:04.328181 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.328157    4559 generic.go:356] "Generic (PLEG): container finished" podID="7498470d-5bab-4716-8feb-c0e70788cc88" containerID="041ee35467d0f895bb50a242fb2ab3a38e29f7fc2bf259739e0de2206cf80a17" exitCode=60
Oct 29 23:00:04.328329 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.328229    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd" event={"ID":"7498470d-5bab-4716-8feb-c0e70788cc88","Type":"ContainerDied","Data":"041ee35467d0f895bb50a242fb2ab3a38e29f7fc2bf259739e0de2206cf80a17"}
Oct 29 23:00:04.328554 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.328530    4559 scope.go:122] "RemoveContainer" containerID="041ee35467d0f895bb50a242fb2ab3a38e29f7fc2bf259739e0de2206cf80a17"
Oct 29 23:00:04.331399 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.331288    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg" event={"ID":"bb6a5323-5dab-421e-9c50-b28825aaa895","Type":"ContainerStarted","Data":"c55701d20b56244e348e9cd27932c77c08cd7482b93de8a58e127b37480182c7"}
Oct 29 23:00:04.337946 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.336923    4559 generic.go:356] "Generic (PLEG): container finished" podID="0fb88948-9bd8-4690-bb16-0e4e5cebb1e3" containerID="211e21d4c283e4441fb7d6d02699d10166bcb6c874096c9698a152611f0fd716" exitCode=60
Oct 29 23:00:04.337946 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.337030    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw" event={"ID":"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3","Type":"ContainerDied","Data":"211e21d4c283e4441fb7d6d02699d10166bcb6c874096c9698a152611f0fd716"}
Oct 29 23:00:04.337946 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.337449    4559 scope.go:122] "RemoveContainer" containerID="211e21d4c283e4441fb7d6d02699d10166bcb6c874096c9698a152611f0fd716"
Oct 29 23:00:04.342677 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.341718    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr" event={"ID":"624b0508-b13d-41aa-977f-8691e3b0ca01","Type":"ContainerStarted","Data":"71f34046432cc4a9423c1a5bf4d5cf98c0df47539d5befe9e768ba0516c18b77"}
Oct 29 23:00:04.344317 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.344287    4559 generic.go:356] "Generic (PLEG): container finished" podID="a2e53e77-e3c5-4468-b830-79e768275dde" containerID="ec1338da3b00de30b953c5dc321ee3d8087d4771bb90d4c1db304cc521b7eafb" exitCode=60
Oct 29 23:00:04.344472 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.344376    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4" event={"ID":"a2e53e77-e3c5-4468-b830-79e768275dde","Type":"ContainerDied","Data":"ec1338da3b00de30b953c5dc321ee3d8087d4771bb90d4c1db304cc521b7eafb"}
Oct 29 23:00:04.344656 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.344633    4559 scope.go:122] "RemoveContainer" containerID="ec1338da3b00de30b953c5dc321ee3d8087d4771bb90d4c1db304cc521b7eafb"
Oct 29 23:00:04.358456 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.358422    4559 generic.go:356] "Generic (PLEG): container finished" podID="58b25906-470a-4dbd-b43a-0f0bcd25223c" containerID="0a72289948f608497898049679c5781981ba7c01aa4a82a7a32f3ee113e8754c" exitCode=60
Oct 29 23:00:04.358629 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.358529    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7" event={"ID":"58b25906-470a-4dbd-b43a-0f0bcd25223c","Type":"ContainerDied","Data":"0a72289948f608497898049679c5781981ba7c01aa4a82a7a32f3ee113e8754c"}
Oct 29 23:00:04.359011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.358986    4559 scope.go:122] "RemoveContainer" containerID="0a72289948f608497898049679c5781981ba7c01aa4a82a7a32f3ee113e8754c"
Oct 29 23:00:04.364464 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.364405    4559 generic.go:356] "Generic (PLEG): container finished" podID="32ea5735-9ca9-4980-8031-44fe6c4269ef" containerID="fbe07017ed7b1273b8189a077e69c2dd1edf6069064654c47e85209dc8d4130e" exitCode=60
Oct 29 23:00:04.364667 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.364642    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr" event={"ID":"32ea5735-9ca9-4980-8031-44fe6c4269ef","Type":"ContainerDied","Data":"fbe07017ed7b1273b8189a077e69c2dd1edf6069064654c47e85209dc8d4130e"}
Oct 29 23:00:04.365650 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.365621    4559 scope.go:122] "RemoveContainer" containerID="fbe07017ed7b1273b8189a077e69c2dd1edf6069064654c47e85209dc8d4130e"
Oct 29 23:00:04.374190 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.374154    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw" event={"ID":"619b371d-35a7-43d9-ace8-cfc478f2fdfa","Type":"ContainerStarted","Data":"138f1dabf63e5f270fce1c6715296d60456f9313259286208dbe5a03388df6aa"}
Oct 29 23:00:04.380485 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.380455    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8" event={"ID":"64ae42cd-1e69-44b6-b728-fb34e15e46c7","Type":"ContainerStarted","Data":"74b13009c1a8158dbf96aa1d590aa0e0ab18c45e5c2517b0864b72a3f0987597"}
Oct 29 23:00:04.385800 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.385767    4559 generic.go:356] "Generic (PLEG): container finished" podID="40c6ad42-57b2-41c2-a6a5-30fa68a808d4" containerID="7bf0c1e24d8fdcade31a317043042ef1d87e099ba88f89b44f6bc0bad5150941" exitCode=60
Oct 29 23:00:04.386000 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.385900    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4" event={"ID":"40c6ad42-57b2-41c2-a6a5-30fa68a808d4","Type":"ContainerDied","Data":"7bf0c1e24d8fdcade31a317043042ef1d87e099ba88f89b44f6bc0bad5150941"}
Oct 29 23:00:04.386304 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.386278    4559 scope.go:122] "RemoveContainer" containerID="7bf0c1e24d8fdcade31a317043042ef1d87e099ba88f89b44f6bc0bad5150941"
Oct 29 23:00:04.390055 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.390027    4559 generic.go:356] "Generic (PLEG): container finished" podID="f6fdff4a-3510-4774-8c07-dddad4aa7219" containerID="eb53db022d8df0b2a92c819b7b75f2cb58fd04ea834e1eab144a9b3f9de10f81" exitCode=60
Oct 29 23:00:04.390213 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.390117    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt" event={"ID":"f6fdff4a-3510-4774-8c07-dddad4aa7219","Type":"ContainerDied","Data":"eb53db022d8df0b2a92c819b7b75f2cb58fd04ea834e1eab144a9b3f9de10f81"}
Oct 29 23:00:04.390674 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.390651    4559 scope.go:122] "RemoveContainer" containerID="eb53db022d8df0b2a92c819b7b75f2cb58fd04ea834e1eab144a9b3f9de10f81"
Oct 29 23:00:04.725004 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.724927    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8" podStartSLOduration=4.724900643 podStartE2EDuration="4.724900643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:04.688647174 +0000 UTC m=+2743.600090749" watchObservedRunningTime="2026-10-29 23:00:04.724900643 +0000 UTC m=+2743.636344261"
Oct 29 23:00:04.789725 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:04.782800    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw" podStartSLOduration=4.782777503 podStartE2EDuration="4.782777503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:04.742201784 +0000 UTC m=+2743.653645338" watchObservedRunningTime="2026-10-29 23:00:04.782777503 +0000 UTC m=+2743.694221058"
Oct 29 23:00:05.402178 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.402127    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g" event={"ID":"3b1c42ea-be0a-4b54-b41c-cd7116493b54","Type":"ContainerStarted","Data":"da527300b95e84de07189dfc3ad3031f0ff00f186e8a4fa556869878e94322c8"}
Oct 29 23:00:05.404478 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.404448    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd" event={"ID":"7498470d-5bab-4716-8feb-c0e70788cc88","Type":"ContainerStarted","Data":"e594e044379c446d9a1b8ea8ad75f9d90ad17a900c1c8eb00bd967e20fa7e061"}
Oct 29 23:00:05.406617 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.406589    4559 generic.go:356] "Generic (PLEG): container finished" podID="bb6a5323-5dab-421e-9c50-b28825aaa895" containerID="c55701d20b56244e348e9cd27932c77c08cd7482b93de8a58e127b37480182c7" exitCode=60
Oct 29 23:00:05.406742 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.406669    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg" event={"ID":"bb6a5323-5dab-421e-9c50-b28825aaa895","Type":"ContainerDied","Data":"c55701d20b56244e348e9cd27932c77c08cd7482b93de8a58e127b37480182c7"}
Oct 29 23:00:05.406742 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.406722    4559 scope.go:122] "RemoveContainer" containerID="66762401ead96d9624811aef403ede11b3ea2efa9fb08e7cb0bc5d86a0fe40a0"
Oct 29 23:00:05.406940 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.406922    4559 scope.go:122] "RemoveContainer" containerID="c55701d20b56244e348e9cd27932c77c08cd7482b93de8a58e127b37480182c7"
Oct 29 23:00:05.407333 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:05.407311    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg_odh-ai-gateway-infra(bb6a5323-5dab-421e-9c50-b28825aaa895)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg" podUID="bb6a5323-5dab-421e-9c50-b28825aaa895"
Oct 29 23:00:05.409165 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.409138    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw" event={"ID":"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3","Type":"ContainerStarted","Data":"8e2bc4d8c97142af58c96430a313e05785f84a3aafa55f6ef44338aa2fbaff9d"}
Oct 29 23:00:05.411320 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.411295    4559 generic.go:356] "Generic (PLEG): container finished" podID="624b0508-b13d-41aa-977f-8691e3b0ca01" containerID="71f34046432cc4a9423c1a5bf4d5cf98c0df47539d5befe9e768ba0516c18b77" exitCode=60
Oct 29 23:00:05.411453 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.411383    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr" event={"ID":"624b0508-b13d-41aa-977f-8691e3b0ca01","Type":"ContainerDied","Data":"71f34046432cc4a9423c1a5bf4d5cf98c0df47539d5befe9e768ba0516c18b77"}
Oct 29 23:00:05.411671 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.411645    4559 scope.go:122] "RemoveContainer" containerID="71f34046432cc4a9423c1a5bf4d5cf98c0df47539d5befe9e768ba0516c18b77"
Oct 29 23:00:05.412068 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:05.412048    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr_odh-ai-gateway-infra(624b0508-b13d-41aa-977f-8691e3b0ca01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr" podUID="624b0508-b13d-41aa-977f-8691e3b0ca01"
Oct 29 23:00:05.414175 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.414150    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4" event={"ID":"a2e53e77-e3c5-4468-b830-79e768275dde","Type":"ContainerStarted","Data":"ae7fe318830c58ba690101185728d0f1e68abcad3a794d354d1681895f87066d"}
Oct 29 23:00:05.416558 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.416533    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7" event={"ID":"58b25906-470a-4dbd-b43a-0f0bcd25223c","Type":"ContainerStarted","Data":"8c2dbe8947b74663e5c6ead9cc790c2c85b4d554b7c81a9fd1eae5ee77fa7a3d"}
Oct 29 23:00:05.418779 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.418756    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr" event={"ID":"32ea5735-9ca9-4980-8031-44fe6c4269ef","Type":"ContainerStarted","Data":"b870a407077de7f8b5e191591c91014ae270d68524f2f3b55dc6f97308e2d171"}
Oct 29 23:00:05.427529 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.427493    4559 scope.go:122] "RemoveContainer" containerID="9219d6b9e5593ab08156b574e8445b4733bf30738834aabc61153450f8418f1c"
Oct 29 23:00:05.427788 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.427694    4559 generic.go:356] "Generic (PLEG): container finished" podID="619b371d-35a7-43d9-ace8-cfc478f2fdfa" containerID="138f1dabf63e5f270fce1c6715296d60456f9313259286208dbe5a03388df6aa" exitCode=60
Oct 29 23:00:05.427965 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.427878    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw" event={"ID":"619b371d-35a7-43d9-ace8-cfc478f2fdfa","Type":"ContainerDied","Data":"138f1dabf63e5f270fce1c6715296d60456f9313259286208dbe5a03388df6aa"}
Oct 29 23:00:05.428356 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.428330    4559 scope.go:122] "RemoveContainer" containerID="138f1dabf63e5f270fce1c6715296d60456f9313259286208dbe5a03388df6aa"
Oct 29 23:00:05.428825 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:05.428782    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw_odh-ai-gateway-infra(619b371d-35a7-43d9-ace8-cfc478f2fdfa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw" podUID="619b371d-35a7-43d9-ace8-cfc478f2fdfa"
Oct 29 23:00:05.430464 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.430441    4559 generic.go:356] "Generic (PLEG): container finished" podID="64ae42cd-1e69-44b6-b728-fb34e15e46c7" containerID="74b13009c1a8158dbf96aa1d590aa0e0ab18c45e5c2517b0864b72a3f0987597" exitCode=60
Oct 29 23:00:05.430577 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.430512    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8" event={"ID":"64ae42cd-1e69-44b6-b728-fb34e15e46c7","Type":"ContainerDied","Data":"74b13009c1a8158dbf96aa1d590aa0e0ab18c45e5c2517b0864b72a3f0987597"}
Oct 29 23:00:05.430873 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.430855    4559 scope.go:122] "RemoveContainer" containerID="74b13009c1a8158dbf96aa1d590aa0e0ab18c45e5c2517b0864b72a3f0987597"
Oct 29 23:00:05.431242 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:05.431217    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8_odh-ai-gateway-infra(64ae42cd-1e69-44b6-b728-fb34e15e46c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8" podUID="64ae42cd-1e69-44b6-b728-fb34e15e46c7"
Oct 29 23:00:05.432704 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.432680    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4" event={"ID":"40c6ad42-57b2-41c2-a6a5-30fa68a808d4","Type":"ContainerStarted","Data":"ba02dc7df266d560eaa4862fc4e7cd73810bc221137f2659c84daadb774131fe"}
Oct 29 23:00:05.436561 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.436526    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt" event={"ID":"f6fdff4a-3510-4774-8c07-dddad4aa7219","Type":"ContainerStarted","Data":"805a262c5214c847dc9f4c53a0407903bcc5a8e365224fe4c5c0bb893957a962"}
Oct 29 23:00:05.447906 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.447736    4559 scope.go:122] "RemoveContainer" containerID="2569d5e7307b5b4b56240adc5de9dadb54adfd74bb4d0f2eac9de75d162d5124"
Oct 29 23:00:05.468061 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:05.468015    4559 scope.go:122] "RemoveContainer" containerID="4a410020d1d395dd309d754613916224bdef4abb18f211009c0d426e81026bba"
Oct 29 23:00:06.449692 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.449658    4559 scope.go:122] "RemoveContainer" containerID="71f34046432cc4a9423c1a5bf4d5cf98c0df47539d5befe9e768ba0516c18b77"
Oct 29 23:00:06.450262 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:06.450236    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr_odh-ai-gateway-infra(624b0508-b13d-41aa-977f-8691e3b0ca01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr" podUID="624b0508-b13d-41aa-977f-8691e3b0ca01"
Oct 29 23:00:06.451744 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.451719    4559 generic.go:356] "Generic (PLEG): container finished" podID="a2e53e77-e3c5-4468-b830-79e768275dde" containerID="ae7fe318830c58ba690101185728d0f1e68abcad3a794d354d1681895f87066d" exitCode=60
Oct 29 23:00:06.451904 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.451824    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4" event={"ID":"a2e53e77-e3c5-4468-b830-79e768275dde","Type":"ContainerDied","Data":"ae7fe318830c58ba690101185728d0f1e68abcad3a794d354d1681895f87066d"}
Oct 29 23:00:06.451904 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.451895    4559 scope.go:122] "RemoveContainer" containerID="ec1338da3b00de30b953c5dc321ee3d8087d4771bb90d4c1db304cc521b7eafb"
Oct 29 23:00:06.452165 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.452148    4559 scope.go:122] "RemoveContainer" containerID="ae7fe318830c58ba690101185728d0f1e68abcad3a794d354d1681895f87066d"
Oct 29 23:00:06.452596 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:06.452570    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4_odh-ai-gateway-infra(a2e53e77-e3c5-4468-b830-79e768275dde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4" podUID="a2e53e77-e3c5-4468-b830-79e768275dde"
Oct 29 23:00:06.454913 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.454890    4559 generic.go:356] "Generic (PLEG): container finished" podID="58b25906-470a-4dbd-b43a-0f0bcd25223c" containerID="8c2dbe8947b74663e5c6ead9cc790c2c85b4d554b7c81a9fd1eae5ee77fa7a3d" exitCode=60
Oct 29 23:00:06.455022 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.454973    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7" event={"ID":"58b25906-470a-4dbd-b43a-0f0bcd25223c","Type":"ContainerDied","Data":"8c2dbe8947b74663e5c6ead9cc790c2c85b4d554b7c81a9fd1eae5ee77fa7a3d"}
Oct 29 23:00:06.455342 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.455326    4559 scope.go:122] "RemoveContainer" containerID="8c2dbe8947b74663e5c6ead9cc790c2c85b4d554b7c81a9fd1eae5ee77fa7a3d"
Oct 29 23:00:06.455775 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:06.455747    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7_odh-ai-gateway-infra(58b25906-470a-4dbd-b43a-0f0bcd25223c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7" podUID="58b25906-470a-4dbd-b43a-0f0bcd25223c"
Oct 29 23:00:06.457662 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.457631    4559 generic.go:356] "Generic (PLEG): container finished" podID="32ea5735-9ca9-4980-8031-44fe6c4269ef" containerID="b870a407077de7f8b5e191591c91014ae270d68524f2f3b55dc6f97308e2d171" exitCode=60
Oct 29 23:00:06.457761 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.457711    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr" event={"ID":"32ea5735-9ca9-4980-8031-44fe6c4269ef","Type":"ContainerDied","Data":"b870a407077de7f8b5e191591c91014ae270d68524f2f3b55dc6f97308e2d171"}
Oct 29 23:00:06.458183 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.458165    4559 scope.go:122] "RemoveContainer" containerID="b870a407077de7f8b5e191591c91014ae270d68524f2f3b55dc6f97308e2d171"
Oct 29 23:00:06.458566 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:06.458543    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr_odh-ai-gateway-infra(32ea5735-9ca9-4980-8031-44fe6c4269ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr" podUID="32ea5735-9ca9-4980-8031-44fe6c4269ef"
Oct 29 23:00:06.461067 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.461048    4559 scope.go:122] "RemoveContainer" containerID="138f1dabf63e5f270fce1c6715296d60456f9313259286208dbe5a03388df6aa"
Oct 29 23:00:06.461565 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:06.461532    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw_odh-ai-gateway-infra(619b371d-35a7-43d9-ace8-cfc478f2fdfa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw" podUID="619b371d-35a7-43d9-ace8-cfc478f2fdfa"
Oct 29 23:00:06.464372 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.464350    4559 scope.go:122] "RemoveContainer" containerID="74b13009c1a8158dbf96aa1d590aa0e0ab18c45e5c2517b0864b72a3f0987597"
Oct 29 23:00:06.464728 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:06.464702    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8_odh-ai-gateway-infra(64ae42cd-1e69-44b6-b728-fb34e15e46c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8" podUID="64ae42cd-1e69-44b6-b728-fb34e15e46c7"
Oct 29 23:00:06.466093 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.466071    4559 generic.go:356] "Generic (PLEG): container finished" podID="40c6ad42-57b2-41c2-a6a5-30fa68a808d4" containerID="ba02dc7df266d560eaa4862fc4e7cd73810bc221137f2659c84daadb774131fe" exitCode=60
Oct 29 23:00:06.466211 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.466097    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4" event={"ID":"40c6ad42-57b2-41c2-a6a5-30fa68a808d4","Type":"ContainerDied","Data":"ba02dc7df266d560eaa4862fc4e7cd73810bc221137f2659c84daadb774131fe"}
Oct 29 23:00:06.466464 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.466446    4559 scope.go:122] "RemoveContainer" containerID="ba02dc7df266d560eaa4862fc4e7cd73810bc221137f2659c84daadb774131fe"
Oct 29 23:00:06.466845 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:06.466796    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4_odh-ai-gateway-infra(40c6ad42-57b2-41c2-a6a5-30fa68a808d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4" podUID="40c6ad42-57b2-41c2-a6a5-30fa68a808d4"
Oct 29 23:00:06.468237 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.468182    4559 generic.go:356] "Generic (PLEG): container finished" podID="f6fdff4a-3510-4774-8c07-dddad4aa7219" containerID="805a262c5214c847dc9f4c53a0407903bcc5a8e365224fe4c5c0bb893957a962" exitCode=60
Oct 29 23:00:06.468349 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.468263    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt" event={"ID":"f6fdff4a-3510-4774-8c07-dddad4aa7219","Type":"ContainerDied","Data":"805a262c5214c847dc9f4c53a0407903bcc5a8e365224fe4c5c0bb893957a962"}
Oct 29 23:00:06.468539 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.468521    4559 scope.go:122] "RemoveContainer" containerID="805a262c5214c847dc9f4c53a0407903bcc5a8e365224fe4c5c0bb893957a962"
Oct 29 23:00:06.468905 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:06.468876    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888580-bpvzt_odh-ai-gateway-infra(f6fdff4a-3510-4774-8c07-dddad4aa7219)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt" podUID="f6fdff4a-3510-4774-8c07-dddad4aa7219"
Oct 29 23:00:06.469804 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.469787    4559 scope.go:122] "RemoveContainer" containerID="0a72289948f608497898049679c5781981ba7c01aa4a82a7a32f3ee113e8754c"
Oct 29 23:00:06.471020 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.470999    4559 generic.go:356] "Generic (PLEG): container finished" podID="3b1c42ea-be0a-4b54-b41c-cd7116493b54" containerID="da527300b95e84de07189dfc3ad3031f0ff00f186e8a4fa556869878e94322c8" exitCode=60
Oct 29 23:00:06.471103 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.471073    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g" event={"ID":"3b1c42ea-be0a-4b54-b41c-cd7116493b54","Type":"ContainerDied","Data":"da527300b95e84de07189dfc3ad3031f0ff00f186e8a4fa556869878e94322c8"}
Oct 29 23:00:06.471360 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.471341    4559 scope.go:122] "RemoveContainer" containerID="da527300b95e84de07189dfc3ad3031f0ff00f186e8a4fa556869878e94322c8"
Oct 29 23:00:06.471675 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:06.471655    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g_odh-ai-gateway-infra(3b1c42ea-be0a-4b54-b41c-cd7116493b54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g" podUID="3b1c42ea-be0a-4b54-b41c-cd7116493b54"
Oct 29 23:00:06.474038 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.474020    4559 generic.go:356] "Generic (PLEG): container finished" podID="7498470d-5bab-4716-8feb-c0e70788cc88" containerID="e594e044379c446d9a1b8ea8ad75f9d90ad17a900c1c8eb00bd967e20fa7e061" exitCode=60
Oct 29 23:00:06.474133 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.474096    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd" event={"ID":"7498470d-5bab-4716-8feb-c0e70788cc88","Type":"ContainerDied","Data":"e594e044379c446d9a1b8ea8ad75f9d90ad17a900c1c8eb00bd967e20fa7e061"}
Oct 29 23:00:06.474307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.474293    4559 scope.go:122] "RemoveContainer" containerID="e594e044379c446d9a1b8ea8ad75f9d90ad17a900c1c8eb00bd967e20fa7e061"
Oct 29 23:00:06.474594 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:06.474567    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd_odh-ai-gateway-infra(7498470d-5bab-4716-8feb-c0e70788cc88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd" podUID="7498470d-5bab-4716-8feb-c0e70788cc88"
Oct 29 23:00:06.476362 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.476342    4559 scope.go:122] "RemoveContainer" containerID="c55701d20b56244e348e9cd27932c77c08cd7482b93de8a58e127b37480182c7"
Oct 29 23:00:06.476642 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:06.476621    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg_odh-ai-gateway-infra(bb6a5323-5dab-421e-9c50-b28825aaa895)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg" podUID="bb6a5323-5dab-421e-9c50-b28825aaa895"
Oct 29 23:00:06.478154 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.478132    4559 generic.go:356] "Generic (PLEG): container finished" podID="0fb88948-9bd8-4690-bb16-0e4e5cebb1e3" containerID="8e2bc4d8c97142af58c96430a313e05785f84a3aafa55f6ef44338aa2fbaff9d" exitCode=60
Oct 29 23:00:06.478248 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.478158    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw" event={"ID":"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3","Type":"ContainerDied","Data":"8e2bc4d8c97142af58c96430a313e05785f84a3aafa55f6ef44338aa2fbaff9d"}
Oct 29 23:00:06.478471 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.478455    4559 scope.go:122] "RemoveContainer" containerID="8e2bc4d8c97142af58c96430a313e05785f84a3aafa55f6ef44338aa2fbaff9d"
Oct 29 23:00:06.478784 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:06.478762    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw_odh-ai-gateway-infra(0fb88948-9bd8-4690-bb16-0e4e5cebb1e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw" podUID="0fb88948-9bd8-4690-bb16-0e4e5cebb1e3"
Oct 29 23:00:06.486754 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.486733    4559 scope.go:122] "RemoveContainer" containerID="fbe07017ed7b1273b8189a077e69c2dd1edf6069064654c47e85209dc8d4130e"
Oct 29 23:00:06.503285 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.503260    4559 scope.go:122] "RemoveContainer" containerID="7bf0c1e24d8fdcade31a317043042ef1d87e099ba88f89b44f6bc0bad5150941"
Oct 29 23:00:06.518388 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.518360    4559 scope.go:122] "RemoveContainer" containerID="eb53db022d8df0b2a92c819b7b75f2cb58fd04ea834e1eab144a9b3f9de10f81"
Oct 29 23:00:06.539025 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.538996    4559 scope.go:122] "RemoveContainer" containerID="7f4d6c5904a39a24ce521328393ef25b9b5c7e9a7dd4cc0beafa7d341d0fa829"
Oct 29 23:00:06.555971 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.555940    4559 scope.go:122] "RemoveContainer" containerID="041ee35467d0f895bb50a242fb2ab3a38e29f7fc2bf259739e0de2206cf80a17"
Oct 29 23:00:06.576026 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:06.575995    4559 scope.go:122] "RemoveContainer" containerID="211e21d4c283e4441fb7d6d02699d10166bcb6c874096c9698a152611f0fd716"
Oct 29 23:00:07.538932 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:07.538897    4559 scope.go:122] "RemoveContainer" containerID="ba02dc7df266d560eaa4862fc4e7cd73810bc221137f2659c84daadb774131fe"
Oct 29 23:00:07.539490 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:07.539373    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4_odh-ai-gateway-infra(40c6ad42-57b2-41c2-a6a5-30fa68a808d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4" podUID="40c6ad42-57b2-41c2-a6a5-30fa68a808d4"
Oct 29 23:00:07.539736 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:07.539710    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"]
Oct 29 23:00:07.540111 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:07.540082    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl" podUID="bde5a64f-e410-46b8-8e6f-c2933360770a" containerName="istio-proxy" containerID="cri-o://9ea104efd748881e5a4ff0e329b2447f90218ad74d4f0fc5db7e2cda9a6813e3" gracePeriod=30
Oct 29 23:00:07.544413 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:07.544381    4559 scope.go:122] "RemoveContainer" containerID="805a262c5214c847dc9f4c53a0407903bcc5a8e365224fe4c5c0bb893957a962"
Oct 29 23:00:07.544980 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:07.544924    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888580-bpvzt_odh-ai-gateway-infra(f6fdff4a-3510-4774-8c07-dddad4aa7219)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt" podUID="f6fdff4a-3510-4774-8c07-dddad4aa7219"
Oct 29 23:00:07.551783 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:07.550379    4559 scope.go:122] "RemoveContainer" containerID="da527300b95e84de07189dfc3ad3031f0ff00f186e8a4fa556869878e94322c8"
Oct 29 23:00:07.551783 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:07.550953    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g_odh-ai-gateway-infra(3b1c42ea-be0a-4b54-b41c-cd7116493b54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g" podUID="3b1c42ea-be0a-4b54-b41c-cd7116493b54"
Oct 29 23:00:07.573303 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:07.552948    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"]
Oct 29 23:00:07.573303 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:07.553228    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv" podUID="064c3a9c-29ac-48fa-9382-1a8ac20fb18a" containerName="istio-proxy" containerID="cri-o://790a0605487c3f254abcc23b640adc592e74ee95ac7f6827d7f5aa466b12e912" gracePeriod=30
Oct 29 23:00:07.573303 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:07.558886    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"]
Oct 29 23:00:07.573303 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:07.559103    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd" podUID="84e86523-8b8b-446d-a5de-25172d07da2d" containerName="istio-proxy" containerID="cri-o://8f51657188e0a12a14e3e6b10ebcb1b1639e4671e03e599b7860e09ee82a10d8" gracePeriod=30
Oct 29 23:00:07.573303 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:07.559736    4559 scope.go:122] "RemoveContainer" containerID="e594e044379c446d9a1b8ea8ad75f9d90ad17a900c1c8eb00bd967e20fa7e061"
Oct 29 23:00:07.573303 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:07.565536    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd_odh-ai-gateway-infra(7498470d-5bab-4716-8feb-c0e70788cc88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd" podUID="7498470d-5bab-4716-8feb-c0e70788cc88"
Oct 29 23:00:07.573303 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:07.566905    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"]
Oct 29 23:00:07.573303 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:07.567125    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc" podUID="fed652a5-91ae-4f75-975a-2d858e4c3bb9" containerName="istio-proxy" containerID="cri-o://bad7f05fe49b646480a86cc09be7f0fb9cbd43463dbfe3e201ad207e83bb96ea" gracePeriod=30
Oct 29 23:00:07.573303 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:07.570865    4559 scope.go:122] "RemoveContainer" containerID="8e2bc4d8c97142af58c96430a313e05785f84a3aafa55f6ef44338aa2fbaff9d"
Oct 29 23:00:07.573303 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:07.571241    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw_odh-ai-gateway-infra(0fb88948-9bd8-4690-bb16-0e4e5cebb1e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw" podUID="0fb88948-9bd8-4690-bb16-0e4e5cebb1e3"
Oct 29 23:00:07.584314 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:07.574106    4559 scope.go:122] "RemoveContainer" containerID="ae7fe318830c58ba690101185728d0f1e68abcad3a794d354d1681895f87066d"
Oct 29 23:00:07.584314 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:07.574563    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4_odh-ai-gateway-infra(a2e53e77-e3c5-4468-b830-79e768275dde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4" podUID="a2e53e77-e3c5-4468-b830-79e768275dde"
Oct 29 23:00:07.584314 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:07.577672    4559 scope.go:122] "RemoveContainer" containerID="8c2dbe8947b74663e5c6ead9cc790c2c85b4d554b7c81a9fd1eae5ee77fa7a3d"
Oct 29 23:00:07.584314 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:07.578051    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7_odh-ai-gateway-infra(58b25906-470a-4dbd-b43a-0f0bcd25223c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7" podUID="58b25906-470a-4dbd-b43a-0f0bcd25223c"
Oct 29 23:00:07.584314 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:07.580839    4559 scope.go:122] "RemoveContainer" containerID="b870a407077de7f8b5e191591c91014ae270d68524f2f3b55dc6f97308e2d171"
Oct 29 23:00:07.584314 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:07.581201    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr_odh-ai-gateway-infra(32ea5735-9ca9-4980-8031-44fe6c4269ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr" podUID="32ea5735-9ca9-4980-8031-44fe6c4269ef"
Oct 29 23:00:08.135900 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.132344    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"]
Oct 29 23:00:08.150220 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.149171    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"
Oct 29 23:00:08.172627 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.164852    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:00:08.242172 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.241357    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"]
Oct 29 23:00:08.262725 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.262688    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl\" (UID: \"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"
Oct 29 23:00:08.262924 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.262897    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl\" (UID: \"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"
Oct 29 23:00:08.262975 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.262938    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4pzm\" (UniqueName: \"kubernetes.io/projected/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-kube-api-access-q4pzm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl\" (UID: \"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"
Oct 29 23:00:08.304522 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.304490    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"]
Oct 29 23:00:08.310051 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.310026    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"
Oct 29 23:00:08.343862 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.341992    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"]
Oct 29 23:00:08.366529 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.366492    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4pzm\" (UniqueName: \"kubernetes.io/projected/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-kube-api-access-q4pzm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl\" (UID: \"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"
Oct 29 23:00:08.366772 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.366608    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wrsm\" (UniqueName: \"kubernetes.io/projected/971d539d-ac5b-4684-8897-eb299ae4c2dc-kube-api-access-7wrsm\") pod \"maas-api-revoke-sub-82385d85-jb9r2\" (UID: \"971d539d-ac5b-4684-8897-eb299ae4c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"
Oct 29 23:00:08.366772 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.366669    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl\" (UID: \"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"
Oct 29 23:00:08.366985 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.366804    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/971d539d-ac5b-4684-8897-eb299ae4c2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82385d85-jb9r2\" (UID: \"971d539d-ac5b-4684-8897-eb299ae4c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"
Oct 29 23:00:08.366985 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.366869    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/971d539d-ac5b-4684-8897-eb299ae4c2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82385d85-jb9r2\" (UID: \"971d539d-ac5b-4684-8897-eb299ae4c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"
Oct 29 23:00:08.366985 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.366913    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl\" (UID: \"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"
Oct 29 23:00:08.368363 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.368334    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl\" (UID: \"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"
Oct 29 23:00:08.377315 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.377277    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl\" (UID: \"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"
Oct 29 23:00:08.388297 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.388214    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4pzm\" (UniqueName: \"kubernetes.io/projected/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-kube-api-access-q4pzm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl\" (UID: \"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"
Oct 29 23:00:08.468348 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.468305    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wrsm\" (UniqueName: \"kubernetes.io/projected/971d539d-ac5b-4684-8897-eb299ae4c2dc-kube-api-access-7wrsm\") pod \"maas-api-revoke-sub-82385d85-jb9r2\" (UID: \"971d539d-ac5b-4684-8897-eb299ae4c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"
Oct 29 23:00:08.468616 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.468574    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/971d539d-ac5b-4684-8897-eb299ae4c2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82385d85-jb9r2\" (UID: \"971d539d-ac5b-4684-8897-eb299ae4c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"
Oct 29 23:00:08.468771 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.468634    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/971d539d-ac5b-4684-8897-eb299ae4c2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82385d85-jb9r2\" (UID: \"971d539d-ac5b-4684-8897-eb299ae4c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"
Oct 29 23:00:08.469341 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.469316    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/971d539d-ac5b-4684-8897-eb299ae4c2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82385d85-jb9r2\" (UID: \"971d539d-ac5b-4684-8897-eb299ae4c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"
Oct 29 23:00:08.471239 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.471214    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/971d539d-ac5b-4684-8897-eb299ae4c2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82385d85-jb9r2\" (UID: \"971d539d-ac5b-4684-8897-eb299ae4c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"
Oct 29 23:00:08.488282 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.488255    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wrsm\" (UniqueName: \"kubernetes.io/projected/971d539d-ac5b-4684-8897-eb299ae4c2dc-kube-api-access-7wrsm\") pod \"maas-api-revoke-sub-82385d85-jb9r2\" (UID: \"971d539d-ac5b-4684-8897-eb299ae4c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"
Oct 29 23:00:08.509279 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.509239    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"
Oct 29 23:00:08.628596 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.628307    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"
Oct 29 23:00:08.847271 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.847237    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"]
Oct 29 23:00:08.848201 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:00:08.848162    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e43c86_7cdb_4f7c_aaa3_48a33f03b2f5.slice/crio-66c4937d613005e3e1d0d49a6106cfab21dd531a86c4f0a7067e9a8e24b3cf56 WatchSource:0}: Error finding container 66c4937d613005e3e1d0d49a6106cfab21dd531a86c4f0a7067e9a8e24b3cf56: Status 404 returned error can't find the container with id 66c4937d613005e3e1d0d49a6106cfab21dd531a86c4f0a7067e9a8e24b3cf56
Oct 29 23:00:08.993527 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:00:08.993497    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971d539d_ac5b_4684_8897_eb299ae4c2dc.slice/crio-0dd6b052cd234cb4ce50998671bb12a3803bef6ba453cda03b4c8c3f0ff3efe7 WatchSource:0}: Error finding container 0dd6b052cd234cb4ce50998671bb12a3803bef6ba453cda03b4c8c3f0ff3efe7: Status 404 returned error can't find the container with id 0dd6b052cd234cb4ce50998671bb12a3803bef6ba453cda03b4c8c3f0ff3efe7
Oct 29 23:00:08.994931 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:08.994906    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"]
Oct 29 23:00:09.629895 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:09.629842    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl" event={"ID":"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5","Type":"ContainerStarted","Data":"2d85e0aecfa1ea2b053a2f64a9aa0468b593dc798e1a1f087e3b48cd7b7b51ca"}
Oct 29 23:00:09.630682 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:09.630654    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl" event={"ID":"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5","Type":"ContainerStarted","Data":"66c4937d613005e3e1d0d49a6106cfab21dd531a86c4f0a7067e9a8e24b3cf56"}
Oct 29 23:00:09.649630 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:09.649584    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2" event={"ID":"971d539d-ac5b-4684-8897-eb299ae4c2dc","Type":"ContainerStarted","Data":"21775d168f0fbd4bdb4efafb9d854f4eea1ea3da3fa9949d2e4bd6ee0d4dbb94"}
Oct 29 23:00:09.649937 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:09.649914    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2" event={"ID":"971d539d-ac5b-4684-8897-eb299ae4c2dc","Type":"ContainerStarted","Data":"0dd6b052cd234cb4ce50998671bb12a3803bef6ba453cda03b4c8c3f0ff3efe7"}
Oct 29 23:00:09.689085 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:09.689027    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl" podStartSLOduration=1.6890120199999998 podStartE2EDuration="1.68901202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:09.688542187 +0000 UTC m=+2748.599985743" watchObservedRunningTime="2026-10-29 23:00:09.68901202 +0000 UTC m=+2748.600455574"
Oct 29 23:00:09.732897 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:09.732841    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2" podStartSLOduration=1.7327957189999998 podStartE2EDuration="1.732795719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:09.731700497 +0000 UTC m=+2748.643144052" watchObservedRunningTime="2026-10-29 23:00:09.732795719 +0000 UTC m=+2748.644239276"
Oct 29 23:00:10.364304 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:10.364158    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"]
Oct 29 23:00:10.364484 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:10.364412    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n" podUID="e824b452-5c7a-4d42-8f86-bf43dd571680" containerName="maas-api" containerID="cri-o://61be77f7d8d7c8b15fbef90d1b8cb248298c02ba5e771db55f79f4449bd6cca8" gracePeriod=30
Oct 29 23:00:11.024992 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.022344    4559 generic.go:356] "Generic (PLEG): container finished" podID="e824b452-5c7a-4d42-8f86-bf43dd571680" containerID="61be77f7d8d7c8b15fbef90d1b8cb248298c02ba5e771db55f79f4449bd6cca8" exitCode=0
Oct 29 23:00:11.024992 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.022524    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n" event={"ID":"e824b452-5c7a-4d42-8f86-bf43dd571680","Type":"ContainerDied","Data":"61be77f7d8d7c8b15fbef90d1b8cb248298c02ba5e771db55f79f4449bd6cca8"}
Oct 29 23:00:11.028231 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.027330    4559 generic.go:356] "Generic (PLEG): container finished" podID="94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5" containerID="2d85e0aecfa1ea2b053a2f64a9aa0468b593dc798e1a1f087e3b48cd7b7b51ca" exitCode=0
Oct 29 23:00:11.028231 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.027469    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl" event={"ID":"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5","Type":"ContainerDied","Data":"2d85e0aecfa1ea2b053a2f64a9aa0468b593dc798e1a1f087e3b48cd7b7b51ca"}
Oct 29 23:00:11.028231 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.028142    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 23:00:11.185102 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.185065    4559 generic.go:356] "Generic (PLEG): container finished" podID="971d539d-ac5b-4684-8897-eb299ae4c2dc" containerID="21775d168f0fbd4bdb4efafb9d854f4eea1ea3da3fa9949d2e4bd6ee0d4dbb94" exitCode=0
Oct 29 23:00:11.185600 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.185253    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2" event={"ID":"971d539d-ac5b-4684-8897-eb299ae4c2dc","Type":"ContainerDied","Data":"21775d168f0fbd4bdb4efafb9d854f4eea1ea3da3fa9949d2e4bd6ee0d4dbb94"}
Oct 29 23:00:11.204184 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.204144    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"]
Oct 29 23:00:11.208087 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.208052    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa3c6-c26zs"]
Oct 29 23:00:11.282271 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.282232    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e824b452-5c7a-4d42-8f86-bf43dd571680-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e824b452-5c7a-4d42-8f86-bf43dd571680-metrics-certs\") pod \"e824b452-5c7a-4d42-8f86-bf43dd571680\" (UID: \"e824b452-5c7a-4d42-8f86-bf43dd571680\") "
Oct 29 23:00:11.287450 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.287249    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e824b452-5c7a-4d42-8f86-bf43dd571680-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e824b452-5c7a-4d42-8f86-bf43dd571680-maas-api-tls\") pod \"e824b452-5c7a-4d42-8f86-bf43dd571680\" (UID: \"e824b452-5c7a-4d42-8f86-bf43dd571680\") "
Oct 29 23:00:11.287773 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.287753    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e824b452-5c7a-4d42-8f86-bf43dd571680-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e824b452-5c7a-4d42-8f86-bf43dd571680-openshift-service-ca\") pod \"e824b452-5c7a-4d42-8f86-bf43dd571680\" (UID: \"e824b452-5c7a-4d42-8f86-bf43dd571680\") "
Oct 29 23:00:11.288204 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.287998    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e824b452-5c7a-4d42-8f86-bf43dd571680-kube-api-access-dp8gb\" (UniqueName: \"kubernetes.io/projected/e824b452-5c7a-4d42-8f86-bf43dd571680-kube-api-access-dp8gb\") pod \"e824b452-5c7a-4d42-8f86-bf43dd571680\" (UID: \"e824b452-5c7a-4d42-8f86-bf43dd571680\") "
Oct 29 23:00:11.296636 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.296579    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e824b452-5c7a-4d42-8f86-bf43dd571680-openshift-service-ca" pod "e824b452-5c7a-4d42-8f86-bf43dd571680" (UID: "e824b452-5c7a-4d42-8f86-bf43dd571680"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:11.299209 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.299174    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e824b452-5c7a-4d42-8f86-bf43dd571680-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:11.300002 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.299791    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e824b452-5c7a-4d42-8f86-bf43dd571680-maas-api-tls" pod "e824b452-5c7a-4d42-8f86-bf43dd571680" (UID: "e824b452-5c7a-4d42-8f86-bf43dd571680"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:11.300780 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.300672    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e824b452-5c7a-4d42-8f86-bf43dd571680-metrics-certs" pod "e824b452-5c7a-4d42-8f86-bf43dd571680" (UID: "e824b452-5c7a-4d42-8f86-bf43dd571680"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:11.303403 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.303375    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e824b452-5c7a-4d42-8f86-bf43dd571680-kube-api-access-dp8gb" pod "e824b452-5c7a-4d42-8f86-bf43dd571680" (UID: "e824b452-5c7a-4d42-8f86-bf43dd571680"). InnerVolumeSpecName "kube-api-access-dp8gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:11.315704 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.315670    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="581e0652-e8ab-402a-b390-acf4906d786b" path="/var/lib/kubelet/pods/581e0652-e8ab-402a-b390-acf4906d786b/volumes"
Oct 29 23:00:11.401230 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.401198    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e824b452-5c7a-4d42-8f86-bf43dd571680-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:11.401490 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.401462    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp8gb\" (UniqueName: \"kubernetes.io/projected/e824b452-5c7a-4d42-8f86-bf43dd571680-kube-api-access-dp8gb\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:11.401490 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:11.401491    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e824b452-5c7a-4d42-8f86-bf43dd571680-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:12.197520 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.197487    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"
Oct 29 23:00:12.198041 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.197494    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n" event={"ID":"e824b452-5c7a-4d42-8f86-bf43dd571680","Type":"ContainerDied","Data":"72a9589ede96f21d6045bab3e40b9803e9fb07db210d91bb13ba6a3682ab7e32"}
Oct 29 23:00:12.198041 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.197630    4559 scope.go:122] "RemoveContainer" containerID="61be77f7d8d7c8b15fbef90d1b8cb248298c02ba5e771db55f79f4449bd6cca8"
Oct 29 23:00:12.275367 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.275318    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"]
Oct 29 23:00:12.284544 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.284511    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa3c6-6569657d5-xmd4n"]
Oct 29 23:00:12.416305 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.416272    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"
Oct 29 23:00:12.489921 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.489848    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"
Oct 29 23:00:12.523664 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.523626    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/971d539d-ac5b-4684-8897-eb299ae4c2dc-kube-api-access-7wrsm\" (UniqueName: \"kubernetes.io/projected/971d539d-ac5b-4684-8897-eb299ae4c2dc-kube-api-access-7wrsm\") pod \"971d539d-ac5b-4684-8897-eb299ae4c2dc\" (UID: \"971d539d-ac5b-4684-8897-eb299ae4c2dc\") "
Oct 29 23:00:12.523948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.523712    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/971d539d-ac5b-4684-8897-eb299ae4c2dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/971d539d-ac5b-4684-8897-eb299ae4c2dc-maas-api-openshift-ca\") pod \"971d539d-ac5b-4684-8897-eb299ae4c2dc\" (UID: \"971d539d-ac5b-4684-8897-eb299ae4c2dc\") "
Oct 29 23:00:12.523948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.523772    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/971d539d-ac5b-4684-8897-eb299ae4c2dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/971d539d-ac5b-4684-8897-eb299ae4c2dc-maas-api-xks-ca\") pod \"971d539d-ac5b-4684-8897-eb299ae4c2dc\" (UID: \"971d539d-ac5b-4684-8897-eb299ae4c2dc\") "
Oct 29 23:00:12.524118 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.524094    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/971d539d-ac5b-4684-8897-eb299ae4c2dc-maas-api-openshift-ca" pod "971d539d-ac5b-4684-8897-eb299ae4c2dc" (UID: "971d539d-ac5b-4684-8897-eb299ae4c2dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:12.524598 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.524580    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/971d539d-ac5b-4684-8897-eb299ae4c2dc-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:12.526013 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.525983    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971d539d-ac5b-4684-8897-eb299ae4c2dc-kube-api-access-7wrsm" pod "971d539d-ac5b-4684-8897-eb299ae4c2dc" (UID: "971d539d-ac5b-4684-8897-eb299ae4c2dc"). InnerVolumeSpecName "kube-api-access-7wrsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:12.526099 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.526006    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/971d539d-ac5b-4684-8897-eb299ae4c2dc-maas-api-xks-ca" pod "971d539d-ac5b-4684-8897-eb299ae4c2dc" (UID: "971d539d-ac5b-4684-8897-eb299ae4c2dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:12.625380 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.625349    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-maas-api-openshift-ca\") pod \"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5\" (UID: \"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5\") "
Oct 29 23:00:12.625604 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.625587    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-kube-api-access-q4pzm\" (UniqueName: \"kubernetes.io/projected/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-kube-api-access-q4pzm\") pod \"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5\" (UID: \"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5\") "
Oct 29 23:00:12.625665 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.625623    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-maas-api-xks-ca\") pod \"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5\" (UID: \"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5\") "
Oct 29 23:00:12.625716 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.625692    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-maas-api-openshift-ca" pod "94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5" (UID: "94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:12.626398 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.626375    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:12.626398 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.626398    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wrsm\" (UniqueName: \"kubernetes.io/projected/971d539d-ac5b-4684-8897-eb299ae4c2dc-kube-api-access-7wrsm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:12.626561 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.626411    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/971d539d-ac5b-4684-8897-eb299ae4c2dc-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:12.627963 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.627941    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-maas-api-xks-ca" pod "94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5" (UID: "94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:12.628154 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.628069    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-kube-api-access-q4pzm" pod "94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5" (UID: "94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5"). InnerVolumeSpecName "kube-api-access-q4pzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:12.729460 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.728872    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4pzm\" (UniqueName: \"kubernetes.io/projected/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-kube-api-access-q4pzm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:12.729460 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.728902    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:12.893552 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.893522    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 23:00:12.965777 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:12.964461    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.059421    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-podinfo\") pod \"bde5a64f-e410-46b8-8e6f-c2933360770a\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") "
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.059484    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-data\") pod \"bde5a64f-e410-46b8-8e6f-c2933360770a\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") "
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.059585    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-envoy\") pod \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") "
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.059619    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istiod-ca-cert\") pod \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") "
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.059689    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-envoy\") pod \"bde5a64f-e410-46b8-8e6f-c2933360770a\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") "
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.059741    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-workload-certs\") pod \"bde5a64f-e410-46b8-8e6f-c2933360770a\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") "
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.059778    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-credential-socket\") pod \"bde5a64f-e410-46b8-8e6f-c2933360770a\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") "
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.059862    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-token\" (UniqueName: \"kubernetes.io/projected/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-token\") pod \"bde5a64f-e410-46b8-8e6f-c2933360770a\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") "
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.059957    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-data\") pod \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") "
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.060008    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-workload-socket\") pod \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") "
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.060048    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-podinfo\") pod \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") "
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.060085    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-workload-certs\") pod \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") "
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.060191    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-token\" (UniqueName: \"kubernetes.io/projected/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-token\") pod \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") "
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.060218    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-credential-socket\") pod \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") "
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.060285    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed652a5-91ae-4f75-975a-2d858e4c3bb9-kube-api-access-zswj4\" (UniqueName: \"kubernetes.io/projected/fed652a5-91ae-4f75-975a-2d858e4c3bb9-kube-api-access-zswj4\") pod \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\" (UID: \"fed652a5-91ae-4f75-975a-2d858e4c3bb9\") "
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.060479    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-workload-socket\") pod \"bde5a64f-e410-46b8-8e6f-c2933360770a\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") "
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.060533    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde5a64f-e410-46b8-8e6f-c2933360770a-kube-api-access-5rggg\" (UniqueName: \"kubernetes.io/projected/bde5a64f-e410-46b8-8e6f-c2933360770a-kube-api-access-5rggg\") pod \"bde5a64f-e410-46b8-8e6f-c2933360770a\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") "
Oct 29 23:00:13.085926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.060566    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bde5a64f-e410-46b8-8e6f-c2933360770a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bde5a64f-e410-46b8-8e6f-c2933360770a-istiod-ca-cert\") pod \"bde5a64f-e410-46b8-8e6f-c2933360770a\" (UID: \"bde5a64f-e410-46b8-8e6f-c2933360770a\") "
Oct 29 23:00:13.094192 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.092035    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-workload-socket" pod "fed652a5-91ae-4f75-975a-2d858e4c3bb9" (UID: "fed652a5-91ae-4f75-975a-2d858e4c3bb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.094192 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.093389    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-data" pod "fed652a5-91ae-4f75-975a-2d858e4c3bb9" (UID: "fed652a5-91ae-4f75-975a-2d858e4c3bb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.108852 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.099917    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-workload-certs" pod "fed652a5-91ae-4f75-975a-2d858e4c3bb9" (UID: "fed652a5-91ae-4f75-975a-2d858e4c3bb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.108852 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.101164    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-workload-certs" pod "bde5a64f-e410-46b8-8e6f-c2933360770a" (UID: "bde5a64f-e410-46b8-8e6f-c2933360770a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.108852 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.105160    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istiod-ca-cert" pod "fed652a5-91ae-4f75-975a-2d858e4c3bb9" (UID: "fed652a5-91ae-4f75-975a-2d858e4c3bb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:13.108852 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.106888    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-workload-socket" pod "bde5a64f-e410-46b8-8e6f-c2933360770a" (UID: "bde5a64f-e410-46b8-8e6f-c2933360770a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.108852 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.107132    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-credential-socket" pod "bde5a64f-e410-46b8-8e6f-c2933360770a" (UID: "bde5a64f-e410-46b8-8e6f-c2933360770a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.108852 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.107459    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-credential-socket" pod "fed652a5-91ae-4f75-975a-2d858e4c3bb9" (UID: "fed652a5-91ae-4f75-975a-2d858e4c3bb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.123743 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.109491    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde5a64f-e410-46b8-8e6f-c2933360770a-istiod-ca-cert" pod "bde5a64f-e410-46b8-8e6f-c2933360770a" (UID: "bde5a64f-e410-46b8-8e6f-c2933360770a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:13.123743 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.111023    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-data" pod "bde5a64f-e410-46b8-8e6f-c2933360770a" (UID: "bde5a64f-e410-46b8-8e6f-c2933360770a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.156407 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.130138    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-token" pod "fed652a5-91ae-4f75-975a-2d858e4c3bb9" (UID: "fed652a5-91ae-4f75-975a-2d858e4c3bb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.156407 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.131054    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-token" pod "bde5a64f-e410-46b8-8e6f-c2933360770a" (UID: "bde5a64f-e410-46b8-8e6f-c2933360770a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.156874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.156802    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde5a64f-e410-46b8-8e6f-c2933360770a-kube-api-access-5rggg" pod "bde5a64f-e410-46b8-8e6f-c2933360770a" (UID: "bde5a64f-e410-46b8-8e6f-c2933360770a"). InnerVolumeSpecName "kube-api-access-5rggg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.157027 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.156990    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-envoy" pod "bde5a64f-e410-46b8-8e6f-c2933360770a" (UID: "bde5a64f-e410-46b8-8e6f-c2933360770a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.157170 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.157144    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-envoy" pod "fed652a5-91ae-4f75-975a-2d858e4c3bb9" (UID: "fed652a5-91ae-4f75-975a-2d858e4c3bb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.157294 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.157273    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-podinfo" pod "fed652a5-91ae-4f75-975a-2d858e4c3bb9" (UID: "fed652a5-91ae-4f75-975a-2d858e4c3bb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:13.157476 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.157452    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed652a5-91ae-4f75-975a-2d858e4c3bb9-kube-api-access-zswj4" pod "fed652a5-91ae-4f75-975a-2d858e4c3bb9" (UID: "fed652a5-91ae-4f75-975a-2d858e4c3bb9"). InnerVolumeSpecName "kube-api-access-zswj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.161933 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.161585    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-podinfo" pod "bde5a64f-e410-46b8-8e6f-c2933360770a" (UID: "bde5a64f-e410-46b8-8e6f-c2933360770a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:13.166108 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166072    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.166108 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166107    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.166346 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166120    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.166346 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166132    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.166346 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166144    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.166346 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166155    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.166346 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166167    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.166346 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166179    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zswj4\" (UniqueName: \"kubernetes.io/projected/fed652a5-91ae-4f75-975a-2d858e4c3bb9-kube-api-access-zswj4\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.166346 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166194    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.166346 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166208    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rggg\" (UniqueName: \"kubernetes.io/projected/bde5a64f-e410-46b8-8e6f-c2933360770a-kube-api-access-5rggg\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.166346 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166220    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bde5a64f-e410-46b8-8e6f-c2933360770a-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.166346 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166231    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.166346 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166244    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.166346 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166254    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.166346 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166264    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed652a5-91ae-4f75-975a-2d858e4c3bb9-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.166346 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166275    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.166346 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166286    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.166346 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166297    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde5a64f-e410-46b8-8e6f-c2933360770a-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.167267 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.166731    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 23:00:13.175783 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.175687    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 23:00:13.210866 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.210765    4559 generic.go:356] "Generic (PLEG): container finished" podID="bde5a64f-e410-46b8-8e6f-c2933360770a" containerID="9ea104efd748881e5a4ff0e329b2447f90218ad74d4f0fc5db7e2cda9a6813e3" exitCode=0
Oct 29 23:00:13.211948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.210870    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl" event={"ID":"bde5a64f-e410-46b8-8e6f-c2933360770a","Type":"ContainerDied","Data":"9ea104efd748881e5a4ff0e329b2447f90218ad74d4f0fc5db7e2cda9a6813e3"}
Oct 29 23:00:13.211948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.210877    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"
Oct 29 23:00:13.211948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.210951    4559 scope.go:122] "RemoveContainer" containerID="9ea104efd748881e5a4ff0e329b2447f90218ad74d4f0fc5db7e2cda9a6813e3"
Oct 29 23:00:13.211948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.210936    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl" event={"ID":"bde5a64f-e410-46b8-8e6f-c2933360770a","Type":"ContainerDied","Data":"1f5b6ac94f56a6eca7b60e3e7137d53b777c574a0766f17aa5a09ef68c808d9c"}
Oct 29 23:00:13.217336 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.213747    4559 generic.go:356] "Generic (PLEG): container finished" podID="fed652a5-91ae-4f75-975a-2d858e4c3bb9" containerID="bad7f05fe49b646480a86cc09be7f0fb9cbd43463dbfe3e201ad207e83bb96ea" exitCode=0
Oct 29 23:00:13.217336 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.214101    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc" event={"ID":"fed652a5-91ae-4f75-975a-2d858e4c3bb9","Type":"ContainerDied","Data":"bad7f05fe49b646480a86cc09be7f0fb9cbd43463dbfe3e201ad207e83bb96ea"}
Oct 29 23:00:13.217336 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.214151    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc" event={"ID":"fed652a5-91ae-4f75-975a-2d858e4c3bb9","Type":"ContainerDied","Data":"8a1d52d654e7fc0488dba916e892bca19c243652f2a489dc958f6833a479827d"}
Oct 29 23:00:13.217336 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.214232    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"
Oct 29 23:00:13.218854 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.218788    4559 generic.go:356] "Generic (PLEG): container finished" podID="064c3a9c-29ac-48fa-9382-1a8ac20fb18a" containerID="790a0605487c3f254abcc23b640adc592e74ee95ac7f6827d7f5aa466b12e912" exitCode=0
Oct 29 23:00:13.219094 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.218916    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv" event={"ID":"064c3a9c-29ac-48fa-9382-1a8ac20fb18a","Type":"ContainerDied","Data":"790a0605487c3f254abcc23b640adc592e74ee95ac7f6827d7f5aa466b12e912"}
Oct 29 23:00:13.219094 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.218977    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv" event={"ID":"064c3a9c-29ac-48fa-9382-1a8ac20fb18a","Type":"ContainerDied","Data":"e6a1d4f565abe2fd0d76c2781b4cec9fecebe6ac3c7b36db2d64945d382dc1c3"}
Oct 29 23:00:13.220605 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.220435    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"
Oct 29 23:00:13.225300 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.225271    4559 generic.go:356] "Generic (PLEG): container finished" podID="84e86523-8b8b-446d-a5de-25172d07da2d" containerID="8f51657188e0a12a14e3e6b10ebcb1b1639e4671e03e599b7860e09ee82a10d8" exitCode=0
Oct 29 23:00:13.225425 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.225392    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"
Oct 29 23:00:13.230396 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.228130    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"
Oct 29 23:00:13.230396 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.228667    4559 scope.go:122] "RemoveContainer" containerID="4a2e8cf6f91c20c461350b7aad28501e7d95c5c1389a463b17cbfb527e920529"
Oct 29 23:00:13.233465 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.233438    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"
Oct 29 23:00:13.248681 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.248643    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e824b452-5c7a-4d42-8f86-bf43dd571680" path="/var/lib/kubelet/pods/e824b452-5c7a-4d42-8f86-bf43dd571680/volumes"
Oct 29 23:00:13.250604 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.250578    4559 scope.go:122] "RemoveContainer" containerID="9ea104efd748881e5a4ff0e329b2447f90218ad74d4f0fc5db7e2cda9a6813e3"
Oct 29 23:00:13.251655 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:13.251619    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea104efd748881e5a4ff0e329b2447f90218ad74d4f0fc5db7e2cda9a6813e3\": container with ID starting with 9ea104efd748881e5a4ff0e329b2447f90218ad74d4f0fc5db7e2cda9a6813e3 not found: ID does not exist" containerID="9ea104efd748881e5a4ff0e329b2447f90218ad74d4f0fc5db7e2cda9a6813e3"
Oct 29 23:00:13.251777 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.251667    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea104efd748881e5a4ff0e329b2447f90218ad74d4f0fc5db7e2cda9a6813e3"} err="failed to get container status \"9ea104efd748881e5a4ff0e329b2447f90218ad74d4f0fc5db7e2cda9a6813e3\": rpc error: code = NotFound desc = could not find container \"9ea104efd748881e5a4ff0e329b2447f90218ad74d4f0fc5db7e2cda9a6813e3\": container with ID starting with 9ea104efd748881e5a4ff0e329b2447f90218ad74d4f0fc5db7e2cda9a6813e3 not found: ID does not exist"
Oct 29 23:00:13.251777 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.251691    4559 scope.go:122] "RemoveContainer" containerID="4a2e8cf6f91c20c461350b7aad28501e7d95c5c1389a463b17cbfb527e920529"
Oct 29 23:00:13.252454 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:13.252125    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a2e8cf6f91c20c461350b7aad28501e7d95c5c1389a463b17cbfb527e920529\": container with ID starting with 4a2e8cf6f91c20c461350b7aad28501e7d95c5c1389a463b17cbfb527e920529 not found: ID does not exist" containerID="4a2e8cf6f91c20c461350b7aad28501e7d95c5c1389a463b17cbfb527e920529"
Oct 29 23:00:13.252454 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.252156    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a2e8cf6f91c20c461350b7aad28501e7d95c5c1389a463b17cbfb527e920529"} err="failed to get container status \"4a2e8cf6f91c20c461350b7aad28501e7d95c5c1389a463b17cbfb527e920529\": rpc error: code = NotFound desc = could not find container \"4a2e8cf6f91c20c461350b7aad28501e7d95c5c1389a463b17cbfb527e920529\": container with ID starting with 4a2e8cf6f91c20c461350b7aad28501e7d95c5c1389a463b17cbfb527e920529 not found: ID does not exist"
Oct 29 23:00:13.252454 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.252180    4559 scope.go:122] "RemoveContainer" containerID="bad7f05fe49b646480a86cc09be7f0fb9cbd43463dbfe3e201ad207e83bb96ea"
Oct 29 23:00:13.267310 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.267274    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-istio-data\") pod \"84e86523-8b8b-446d-a5de-25172d07da2d\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") "
Oct 29 23:00:13.267492 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.267337    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istiod-ca-cert\") pod \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") "
Oct 29 23:00:13.267492 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.267390    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-workload-socket\") pod \"84e86523-8b8b-446d-a5de-25172d07da2d\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") "
Oct 29 23:00:13.267492 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.267424    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e86523-8b8b-446d-a5de-25172d07da2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e86523-8b8b-446d-a5de-25172d07da2d-istiod-ca-cert\") pod \"84e86523-8b8b-446d-a5de-25172d07da2d\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") "
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.267525    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e86523-8b8b-446d-a5de-25172d07da2d-kube-api-access-xc64k\" (UniqueName: \"kubernetes.io/projected/84e86523-8b8b-446d-a5de-25172d07da2d-kube-api-access-xc64k\") pod \"84e86523-8b8b-446d-a5de-25172d07da2d\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") "
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.267553    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-workload-socket\") pod \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") "
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.267594    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-istio-data" pod "84e86523-8b8b-446d-a5de-25172d07da2d" (UID: "84e86523-8b8b-446d-a5de-25172d07da2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.267627    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-envoy\") pod \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") "
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.267661    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84e86523-8b8b-446d-a5de-25172d07da2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e86523-8b8b-446d-a5de-25172d07da2d-istio-podinfo\") pod \"84e86523-8b8b-446d-a5de-25172d07da2d\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") "
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.267691    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-data\") pod \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") "
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.267739    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-istio-envoy\") pod \"84e86523-8b8b-446d-a5de-25172d07da2d\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") "
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.267797    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-workload-certs\") pod \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") "
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.267892    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-workload-certs\") pod \"84e86523-8b8b-446d-a5de-25172d07da2d\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") "
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.267927    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-podinfo\") pod \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") "
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.267960    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e86523-8b8b-446d-a5de-25172d07da2d-istiod-ca-cert" pod "84e86523-8b8b-446d-a5de-25172d07da2d" (UID: "84e86523-8b8b-446d-a5de-25172d07da2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.267967    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-credential-socket\") pod \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") "
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.268273    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-credential-socket" pod "064c3a9c-29ac-48fa-9382-1a8ac20fb18a" (UID: "064c3a9c-29ac-48fa-9382-1a8ac20fb18a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.268306    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e86523-8b8b-446d-a5de-25172d07da2d-istio-token\" (UniqueName: \"kubernetes.io/projected/84e86523-8b8b-446d-a5de-25172d07da2d-istio-token\") pod \"84e86523-8b8b-446d-a5de-25172d07da2d\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") "
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.268365    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-credential-socket\") pod \"84e86523-8b8b-446d-a5de-25172d07da2d\" (UID: \"84e86523-8b8b-446d-a5de-25172d07da2d\") "
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.268404    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-token\" (UniqueName: \"kubernetes.io/projected/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-token\") pod \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") "
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.268474    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-kube-api-access-hjlpx\" (UniqueName: \"kubernetes.io/projected/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-kube-api-access-hjlpx\") pod \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\" (UID: \"064c3a9c-29ac-48fa-9382-1a8ac20fb18a\") "
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.268793    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-data" pod "064c3a9c-29ac-48fa-9382-1a8ac20fb18a" (UID: "064c3a9c-29ac-48fa-9382-1a8ac20fb18a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.269031    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-workload-socket" pod "064c3a9c-29ac-48fa-9382-1a8ac20fb18a" (UID: "064c3a9c-29ac-48fa-9382-1a8ac20fb18a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.269181    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-workload-certs" pod "064c3a9c-29ac-48fa-9382-1a8ac20fb18a" (UID: "064c3a9c-29ac-48fa-9382-1a8ac20fb18a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.269337    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istiod-ca-cert" pod "064c3a9c-29ac-48fa-9382-1a8ac20fb18a" (UID: "064c3a9c-29ac-48fa-9382-1a8ac20fb18a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.269533    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.269557    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.269558    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-workload-socket" pod "84e86523-8b8b-446d-a5de-25172d07da2d" (UID: "84e86523-8b8b-446d-a5de-25172d07da2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.269573    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.269586    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.269599    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.269612    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.270011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.269625    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e86523-8b8b-446d-a5de-25172d07da2d-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.275080 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.274776    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-istio-envoy" pod "84e86523-8b8b-446d-a5de-25172d07da2d" (UID: "84e86523-8b8b-446d-a5de-25172d07da2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.277265 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.277235    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-credential-socket" pod "84e86523-8b8b-446d-a5de-25172d07da2d" (UID: "84e86523-8b8b-446d-a5de-25172d07da2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.280147 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.280069    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-workload-certs" pod "84e86523-8b8b-446d-a5de-25172d07da2d" (UID: "84e86523-8b8b-446d-a5de-25172d07da2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.285541 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.285487    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e86523-8b8b-446d-a5de-25172d07da2d-kube-api-access-xc64k" pod "84e86523-8b8b-446d-a5de-25172d07da2d" (UID: "84e86523-8b8b-446d-a5de-25172d07da2d"). InnerVolumeSpecName "kube-api-access-xc64k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.287600 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.287560    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e86523-8b8b-446d-a5de-25172d07da2d-istio-token" pod "84e86523-8b8b-446d-a5de-25172d07da2d" (UID: "84e86523-8b8b-446d-a5de-25172d07da2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.290681 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.290506    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd" event={"ID":"84e86523-8b8b-446d-a5de-25172d07da2d","Type":"ContainerDied","Data":"8f51657188e0a12a14e3e6b10ebcb1b1639e4671e03e599b7860e09ee82a10d8"}
Oct 29 23:00:13.290681 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.290575    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd" event={"ID":"84e86523-8b8b-446d-a5de-25172d07da2d","Type":"ContainerDied","Data":"0a497f066bf2396e229772f63d58eb4510cb649bdf919668af120f4b4be0b682"}
Oct 29 23:00:13.290681 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.290600    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl" event={"ID":"94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5","Type":"ContainerDied","Data":"66c4937d613005e3e1d0d49a6106cfab21dd531a86c4f0a7067e9a8e24b3cf56"}
Oct 29 23:00:13.290681 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.290615    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66c4937d613005e3e1d0d49a6106cfab21dd531a86c4f0a7067e9a8e24b3cf56"
Oct 29 23:00:13.290681 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.290627    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2" event={"ID":"971d539d-ac5b-4684-8897-eb299ae4c2dc","Type":"ContainerDied","Data":"0dd6b052cd234cb4ce50998671bb12a3803bef6ba453cda03b4c8c3f0ff3efe7"}
Oct 29 23:00:13.290681 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.290649    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dd6b052cd234cb4ce50998671bb12a3803bef6ba453cda03b4c8c3f0ff3efe7"
Oct 29 23:00:13.291146 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.291119    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84e86523-8b8b-446d-a5de-25172d07da2d-istio-podinfo" pod "84e86523-8b8b-446d-a5de-25172d07da2d" (UID: "84e86523-8b8b-446d-a5de-25172d07da2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:13.302183 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.302131    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-podinfo" pod "064c3a9c-29ac-48fa-9382-1a8ac20fb18a" (UID: "064c3a9c-29ac-48fa-9382-1a8ac20fb18a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:13.302941 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.302729    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-kube-api-access-hjlpx" pod "064c3a9c-29ac-48fa-9382-1a8ac20fb18a" (UID: "064c3a9c-29ac-48fa-9382-1a8ac20fb18a"). InnerVolumeSpecName "kube-api-access-hjlpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.302941 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.302898    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-token" pod "064c3a9c-29ac-48fa-9382-1a8ac20fb18a" (UID: "064c3a9c-29ac-48fa-9382-1a8ac20fb18a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.302941 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.302933    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-envoy" pod "064c3a9c-29ac-48fa-9382-1a8ac20fb18a" (UID: "064c3a9c-29ac-48fa-9382-1a8ac20fb18a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.306933 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.306911    4559 scope.go:122] "RemoveContainer" containerID="bad7f05fe49b646480a86cc09be7f0fb9cbd43463dbfe3e201ad207e83bb96ea"
Oct 29 23:00:13.307318 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:13.307291    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad7f05fe49b646480a86cc09be7f0fb9cbd43463dbfe3e201ad207e83bb96ea\": container with ID starting with bad7f05fe49b646480a86cc09be7f0fb9cbd43463dbfe3e201ad207e83bb96ea not found: ID does not exist" containerID="bad7f05fe49b646480a86cc09be7f0fb9cbd43463dbfe3e201ad207e83bb96ea"
Oct 29 23:00:13.307446 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.307331    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad7f05fe49b646480a86cc09be7f0fb9cbd43463dbfe3e201ad207e83bb96ea"} err="failed to get container status \"bad7f05fe49b646480a86cc09be7f0fb9cbd43463dbfe3e201ad207e83bb96ea\": rpc error: code = NotFound desc = could not find container \"bad7f05fe49b646480a86cc09be7f0fb9cbd43463dbfe3e201ad207e83bb96ea\": container with ID starting with bad7f05fe49b646480a86cc09be7f0fb9cbd43463dbfe3e201ad207e83bb96ea not found: ID does not exist"
Oct 29 23:00:13.307446 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.307356    4559 scope.go:122] "RemoveContainer" containerID="790a0605487c3f254abcc23b640adc592e74ee95ac7f6827d7f5aa466b12e912"
Oct 29 23:00:13.319727 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.319701    4559 scope.go:122] "RemoveContainer" containerID="790a0605487c3f254abcc23b640adc592e74ee95ac7f6827d7f5aa466b12e912"
Oct 29 23:00:13.320219 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:13.320193    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790a0605487c3f254abcc23b640adc592e74ee95ac7f6827d7f5aa466b12e912\": container with ID starting with 790a0605487c3f254abcc23b640adc592e74ee95ac7f6827d7f5aa466b12e912 not found: ID does not exist" containerID="790a0605487c3f254abcc23b640adc592e74ee95ac7f6827d7f5aa466b12e912"
Oct 29 23:00:13.320323 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.320230    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790a0605487c3f254abcc23b640adc592e74ee95ac7f6827d7f5aa466b12e912"} err="failed to get container status \"790a0605487c3f254abcc23b640adc592e74ee95ac7f6827d7f5aa466b12e912\": rpc error: code = NotFound desc = could not find container \"790a0605487c3f254abcc23b640adc592e74ee95ac7f6827d7f5aa466b12e912\": container with ID starting with 790a0605487c3f254abcc23b640adc592e74ee95ac7f6827d7f5aa466b12e912 not found: ID does not exist"
Oct 29 23:00:13.320323 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.320252    4559 scope.go:122] "RemoveContainer" containerID="8f51657188e0a12a14e3e6b10ebcb1b1639e4671e03e599b7860e09ee82a10d8"
Oct 29 23:00:13.332565 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.332541    4559 scope.go:122] "RemoveContainer" containerID="8f51657188e0a12a14e3e6b10ebcb1b1639e4671e03e599b7860e09ee82a10d8"
Oct 29 23:00:13.333052 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:13.333026    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f51657188e0a12a14e3e6b10ebcb1b1639e4671e03e599b7860e09ee82a10d8\": container with ID starting with 8f51657188e0a12a14e3e6b10ebcb1b1639e4671e03e599b7860e09ee82a10d8 not found: ID does not exist" containerID="8f51657188e0a12a14e3e6b10ebcb1b1639e4671e03e599b7860e09ee82a10d8"
Oct 29 23:00:13.333164 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.333065    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f51657188e0a12a14e3e6b10ebcb1b1639e4671e03e599b7860e09ee82a10d8"} err="failed to get container status \"8f51657188e0a12a14e3e6b10ebcb1b1639e4671e03e599b7860e09ee82a10d8\": rpc error: code = NotFound desc = could not find container \"8f51657188e0a12a14e3e6b10ebcb1b1639e4671e03e599b7860e09ee82a10d8\": container with ID starting with 8f51657188e0a12a14e3e6b10ebcb1b1639e4671e03e599b7860e09ee82a10d8 not found: ID does not exist"
Oct 29 23:00:13.342399 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.342261    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"]
Oct 29 23:00:13.348920 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.348889    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-mb2dl"]
Oct 29 23:00:13.371784 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.371748    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.371784 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.371785    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.372080 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.371798    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e86523-8b8b-446d-a5de-25172d07da2d-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.372080 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.371830    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.372080 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.371846    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.372080 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.371863    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjlpx\" (UniqueName: \"kubernetes.io/projected/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-kube-api-access-hjlpx\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.372080 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.371877    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.372080 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.371898    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc64k\" (UniqueName: \"kubernetes.io/projected/84e86523-8b8b-446d-a5de-25172d07da2d-kube-api-access-xc64k\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.372080 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.371910    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/064c3a9c-29ac-48fa-9382-1a8ac20fb18a-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.372080 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.371923    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e86523-8b8b-446d-a5de-25172d07da2d-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.372080 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.371936    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e86523-8b8b-446d-a5de-25172d07da2d-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.417469 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.417425    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"]
Oct 29 23:00:13.428753 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.428714    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-x9bxc"]
Oct 29 23:00:13.575645 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.575605    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"]
Oct 29 23:00:13.581401 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.581370    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-n2ttv"]
Oct 29 23:00:13.613991 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.613902    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"]
Oct 29 23:00:13.623590 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:13.623558    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ctqgd"]
Oct 29 23:00:13.696876 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:13.696835    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:00:15.230352 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:15.230315    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="064c3a9c-29ac-48fa-9382-1a8ac20fb18a" path="/var/lib/kubelet/pods/064c3a9c-29ac-48fa-9382-1a8ac20fb18a/volumes"
Oct 29 23:00:15.230916 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:15.230901    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e86523-8b8b-446d-a5de-25172d07da2d" path="/var/lib/kubelet/pods/84e86523-8b8b-446d-a5de-25172d07da2d/volumes"
Oct 29 23:00:15.231386 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:15.231372    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bde5a64f-e410-46b8-8e6f-c2933360770a" path="/var/lib/kubelet/pods/bde5a64f-e410-46b8-8e6f-c2933360770a/volumes"
Oct 29 23:00:15.232206 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:15.232189    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fed652a5-91ae-4f75-975a-2d858e4c3bb9" path="/var/lib/kubelet/pods/fed652a5-91ae-4f75-975a-2d858e4c3bb9/volumes"
Oct 29 23:00:15.233446 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:15.233429    4559 scope.go:122] "RemoveContainer" containerID="138f1dabf63e5f270fce1c6715296d60456f9313259286208dbe5a03388df6aa"
Oct 29 23:00:16.314756 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:16.314720    4559 generic.go:356] "Generic (PLEG): container finished" podID="619b371d-35a7-43d9-ace8-cfc478f2fdfa" containerID="4ee0c062451341ca2d01bb9394394e10abc21dc2cbe98de2ea1b6495da2d4f4b" exitCode=60
Oct 29 23:00:16.315322 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:16.314949    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw" event={"ID":"619b371d-35a7-43d9-ace8-cfc478f2fdfa","Type":"ContainerDied","Data":"4ee0c062451341ca2d01bb9394394e10abc21dc2cbe98de2ea1b6495da2d4f4b"}
Oct 29 23:00:16.315322 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:16.315016    4559 scope.go:122] "RemoveContainer" containerID="138f1dabf63e5f270fce1c6715296d60456f9313259286208dbe5a03388df6aa"
Oct 29 23:00:16.315459 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:16.315439    4559 scope.go:122] "RemoveContainer" containerID="4ee0c062451341ca2d01bb9394394e10abc21dc2cbe98de2ea1b6495da2d4f4b"
Oct 29 23:00:16.315935 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:16.315912    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw_odh-ai-gateway-infra(619b371d-35a7-43d9-ace8-cfc478f2fdfa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw" podUID="619b371d-35a7-43d9-ace8-cfc478f2fdfa"
Oct 29 23:00:17.367492 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:17.367453    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw"]
Oct 29 23:00:17.574887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:17.574798    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw"
Oct 29 23:00:17.664269 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:17.664221    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/619b371d-35a7-43d9-ace8-cfc478f2fdfa-kube-api-access-j4477\" (UniqueName: \"kubernetes.io/projected/619b371d-35a7-43d9-ace8-cfc478f2fdfa-kube-api-access-j4477\") pod \"619b371d-35a7-43d9-ace8-cfc478f2fdfa\" (UID: \"619b371d-35a7-43d9-ace8-cfc478f2fdfa\") "
Oct 29 23:00:17.664887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:17.664859    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/619b371d-35a7-43d9-ace8-cfc478f2fdfa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619b371d-35a7-43d9-ace8-cfc478f2fdfa-maas-api-service-ca\") pod \"619b371d-35a7-43d9-ace8-cfc478f2fdfa\" (UID: \"619b371d-35a7-43d9-ace8-cfc478f2fdfa\") "
Oct 29 23:00:17.666629 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:17.666602    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619b371d-35a7-43d9-ace8-cfc478f2fdfa-maas-api-service-ca" pod "619b371d-35a7-43d9-ace8-cfc478f2fdfa" (UID: "619b371d-35a7-43d9-ace8-cfc478f2fdfa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:17.672121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:17.672070    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619b371d-35a7-43d9-ace8-cfc478f2fdfa-kube-api-access-j4477" pod "619b371d-35a7-43d9-ace8-cfc478f2fdfa" (UID: "619b371d-35a7-43d9-ace8-cfc478f2fdfa"). InnerVolumeSpecName "kube-api-access-j4477". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:17.767414 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:17.767352    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619b371d-35a7-43d9-ace8-cfc478f2fdfa-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:17.767414 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:17.767388    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4477\" (UniqueName: \"kubernetes.io/projected/619b371d-35a7-43d9-ace8-cfc478f2fdfa-kube-api-access-j4477\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:18.223878 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:18.219415    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"]
Oct 29 23:00:18.223878 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:18.221185    4559 scope.go:122] "RemoveContainer" containerID="805a262c5214c847dc9f4c53a0407903bcc5a8e365224fe4c5c0bb893957a962"
Oct 29 23:00:18.223878 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:18.221927    4559 scope.go:122] "RemoveContainer" containerID="8e2bc4d8c97142af58c96430a313e05785f84a3aafa55f6ef44338aa2fbaff9d"
Oct 29 23:00:18.315590 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:18.309727    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"]
Oct 29 23:00:18.346062 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:18.344101    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw" event={"ID":"619b371d-35a7-43d9-ace8-cfc478f2fdfa","Type":"ContainerDied","Data":"a01c4fcd89df91dee1eacc7e2c1842f675c87110d1ad6e7e76a20175200b3d19"}
Oct 29 23:00:18.346062 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:18.344199    4559 scope.go:122] "RemoveContainer" containerID="4ee0c062451341ca2d01bb9394394e10abc21dc2cbe98de2ea1b6495da2d4f4b"
Oct 29 23:00:18.346062 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:18.344385    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw"
Oct 29 23:00:18.451078 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:18.450722    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw"]
Oct 29 23:00:18.459277 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:18.459239    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02720a-29888580-mmjcw"]
Oct 29 23:00:19.333838 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:19.333373    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="619b371d-35a7-43d9-ace8-cfc478f2fdfa" path="/var/lib/kubelet/pods/619b371d-35a7-43d9-ace8-cfc478f2fdfa/volumes"
Oct 29 23:00:19.343876 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:19.338082    4559 scope.go:122] "RemoveContainer" containerID="8c2dbe8947b74663e5c6ead9cc790c2c85b4d554b7c81a9fd1eae5ee77fa7a3d"
Oct 29 23:00:19.368026 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:19.354196    4559 scope.go:122] "RemoveContainer" containerID="c55701d20b56244e348e9cd27932c77c08cd7482b93de8a58e127b37480182c7"
Oct 29 23:00:19.368026 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:19.354969    4559 scope.go:122] "RemoveContainer" containerID="b870a407077de7f8b5e191591c91014ae270d68524f2f3b55dc6f97308e2d171"
Oct 29 23:00:19.368026 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:19.355553    4559 scope.go:122] "RemoveContainer" containerID="ae7fe318830c58ba690101185728d0f1e68abcad3a794d354d1681895f87066d"
Oct 29 23:00:19.375117 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:19.373935    4559 scope.go:122] "RemoveContainer" containerID="e594e044379c446d9a1b8ea8ad75f9d90ad17a900c1c8eb00bd967e20fa7e061"
Oct 29 23:00:19.383910 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:19.382932    4559 scope.go:122] "RemoveContainer" containerID="71f34046432cc4a9423c1a5bf4d5cf98c0df47539d5befe9e768ba0516c18b77"
Oct 29 23:00:19.411836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:19.411191    4559 scope.go:122] "RemoveContainer" containerID="74b13009c1a8158dbf96aa1d590aa0e0ab18c45e5c2517b0864b72a3f0987597"
Oct 29 23:00:19.411836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:19.411713    4559 scope.go:122] "RemoveContainer" containerID="da527300b95e84de07189dfc3ad3031f0ff00f186e8a4fa556869878e94322c8"
Oct 29 23:00:19.483834 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:19.479352    4559 scope.go:122] "RemoveContainer" containerID="ba02dc7df266d560eaa4862fc4e7cd73810bc221137f2659c84daadb774131fe"
Oct 29 23:00:19.775612 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:19.775571    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt" event={"ID":"f6fdff4a-3510-4774-8c07-dddad4aa7219","Type":"ContainerStarted","Data":"cc674d0bc7e03d51a3e48566553c58023e058f3e9895c0036278f2d5c1b257d4"}
Oct 29 23:00:19.776494 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:19.776468    4559 scope.go:122] "RemoveContainer" containerID="cc674d0bc7e03d51a3e48566553c58023e058f3e9895c0036278f2d5c1b257d4"
Oct 29 23:00:19.777515 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:19.777483    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888580-bpvzt_odh-ai-gateway-infra(f6fdff4a-3510-4774-8c07-dddad4aa7219)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt" podUID="f6fdff4a-3510-4774-8c07-dddad4aa7219"
Oct 29 23:00:19.795361 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:19.795320    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw" event={"ID":"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3","Type":"ContainerStarted","Data":"9ba7e3a52bff2d7f98ee857308d09e441e7dde27536bc27c1c593bd45310aa91"}
Oct 29 23:00:19.795996 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:19.795970    4559 scope.go:122] "RemoveContainer" containerID="9ba7e3a52bff2d7f98ee857308d09e441e7dde27536bc27c1c593bd45310aa91"
Oct 29 23:00:19.796715 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:19.796463    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw_odh-ai-gateway-infra(0fb88948-9bd8-4690-bb16-0e4e5cebb1e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw" podUID="0fb88948-9bd8-4690-bb16-0e4e5cebb1e3"
Oct 29 23:00:19.884000 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:19.883870    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"]
Oct 29 23:00:19.884150 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:19.884114    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m" podUID="f479373b-2190-46af-9b6d-ad901981a56b" containerName="istio-proxy" containerID="cri-o://721665c9e550ef6adbdbf3f4bab54654b871141ed3e1b22105189711c92247c7" gracePeriod=30
Oct 29 23:00:20.811967 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.811867    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7" event={"ID":"58b25906-470a-4dbd-b43a-0f0bcd25223c","Type":"ContainerStarted","Data":"329aa3e1dd370ee4fe4fc735d471f0ea0f429900501b70c0e34c31156d06da49"}
Oct 29 23:00:20.814492 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.814460    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr" event={"ID":"32ea5735-9ca9-4980-8031-44fe6c4269ef","Type":"ContainerStarted","Data":"84bd87febf6bcf2b78f3494bd2149a5f8db99063075e70dad47915947f47f62b"}
Oct 29 23:00:20.816862 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.816833    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8" event={"ID":"64ae42cd-1e69-44b6-b728-fb34e15e46c7","Type":"ContainerStarted","Data":"81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf"}
Oct 29 23:00:20.819359 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.819323    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4" event={"ID":"40c6ad42-57b2-41c2-a6a5-30fa68a808d4","Type":"ContainerStarted","Data":"31a99fc78e854bb9fe10d98f23f327355c236702bd33e15dee14ee67102516c5"}
Oct 29 23:00:20.821444 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.821413    4559 generic.go:356] "Generic (PLEG): container finished" podID="f6fdff4a-3510-4774-8c07-dddad4aa7219" containerID="cc674d0bc7e03d51a3e48566553c58023e058f3e9895c0036278f2d5c1b257d4" exitCode=60
Oct 29 23:00:20.821564 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.821481    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt" event={"ID":"f6fdff4a-3510-4774-8c07-dddad4aa7219","Type":"ContainerDied","Data":"cc674d0bc7e03d51a3e48566553c58023e058f3e9895c0036278f2d5c1b257d4"}
Oct 29 23:00:20.821564 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.821535    4559 scope.go:122] "RemoveContainer" containerID="805a262c5214c847dc9f4c53a0407903bcc5a8e365224fe4c5c0bb893957a962"
Oct 29 23:00:20.821961 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.821942    4559 scope.go:122] "RemoveContainer" containerID="cc674d0bc7e03d51a3e48566553c58023e058f3e9895c0036278f2d5c1b257d4"
Oct 29 23:00:20.822354 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:20.822334    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888580-bpvzt_odh-ai-gateway-infra(f6fdff4a-3510-4774-8c07-dddad4aa7219)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt" podUID="f6fdff4a-3510-4774-8c07-dddad4aa7219"
Oct 29 23:00:20.823673 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.823647    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g" event={"ID":"3b1c42ea-be0a-4b54-b41c-cd7116493b54","Type":"ContainerStarted","Data":"4ecc722049625680c59e47c545c82c9030f8048e72cbc56d043a6ca902e0cc95"}
Oct 29 23:00:20.825645 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.825621    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd" event={"ID":"7498470d-5bab-4716-8feb-c0e70788cc88","Type":"ContainerStarted","Data":"bf2d4ee10fb8f88a1b2fb37e8e8e9f2f29dd6e5ce9772b238b572b00b65ef1eb"}
Oct 29 23:00:20.828063 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.828002    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg" event={"ID":"bb6a5323-5dab-421e-9c50-b28825aaa895","Type":"ContainerStarted","Data":"4b978c63699e8ab4ab4ea9f2ed09240271664b8f578990f0dc9414cf32a8e69c"}
Oct 29 23:00:20.830171 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.830149    4559 generic.go:356] "Generic (PLEG): container finished" podID="0fb88948-9bd8-4690-bb16-0e4e5cebb1e3" containerID="9ba7e3a52bff2d7f98ee857308d09e441e7dde27536bc27c1c593bd45310aa91" exitCode=60
Oct 29 23:00:20.830299 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.830218    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw" event={"ID":"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3","Type":"ContainerDied","Data":"9ba7e3a52bff2d7f98ee857308d09e441e7dde27536bc27c1c593bd45310aa91"}
Oct 29 23:00:20.830629 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.830608    4559 scope.go:122] "RemoveContainer" containerID="9ba7e3a52bff2d7f98ee857308d09e441e7dde27536bc27c1c593bd45310aa91"
Oct 29 23:00:20.830975 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:20.830945    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw_odh-ai-gateway-infra(0fb88948-9bd8-4690-bb16-0e4e5cebb1e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw" podUID="0fb88948-9bd8-4690-bb16-0e4e5cebb1e3"
Oct 29 23:00:20.844416 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.844169    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr" event={"ID":"624b0508-b13d-41aa-977f-8691e3b0ca01","Type":"ContainerStarted","Data":"c4e77de649c60c71a3675ec5d6c28f4ab83d89f9eb7c3ae43e4c239cebd15a12"}
Oct 29 23:00:20.847014 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.846991    4559 scope.go:122] "RemoveContainer" containerID="8e2bc4d8c97142af58c96430a313e05785f84a3aafa55f6ef44338aa2fbaff9d"
Oct 29 23:00:20.849033 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.849006    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4" event={"ID":"a2e53e77-e3c5-4468-b830-79e768275dde","Type":"ContainerStarted","Data":"e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc"}
Oct 29 23:00:20.850392 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.850369    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt"]
Oct 29 23:00:20.889152 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:20.889114    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw"]
Oct 29 23:00:21.866331 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.866289    4559 generic.go:356] "Generic (PLEG): container finished" podID="58b25906-470a-4dbd-b43a-0f0bcd25223c" containerID="329aa3e1dd370ee4fe4fc735d471f0ea0f429900501b70c0e34c31156d06da49" exitCode=60
Oct 29 23:00:21.866331 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.866318    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7" event={"ID":"58b25906-470a-4dbd-b43a-0f0bcd25223c","Type":"ContainerDied","Data":"329aa3e1dd370ee4fe4fc735d471f0ea0f429900501b70c0e34c31156d06da49"}
Oct 29 23:00:21.867094 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.866426    4559 scope.go:122] "RemoveContainer" containerID="8c2dbe8947b74663e5c6ead9cc790c2c85b4d554b7c81a9fd1eae5ee77fa7a3d"
Oct 29 23:00:21.867094 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.866931    4559 scope.go:122] "RemoveContainer" containerID="329aa3e1dd370ee4fe4fc735d471f0ea0f429900501b70c0e34c31156d06da49"
Oct 29 23:00:21.867380 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:21.867350    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7_odh-ai-gateway-infra(58b25906-470a-4dbd-b43a-0f0bcd25223c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7" podUID="58b25906-470a-4dbd-b43a-0f0bcd25223c"
Oct 29 23:00:21.869292 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.869258    4559 generic.go:356] "Generic (PLEG): container finished" podID="32ea5735-9ca9-4980-8031-44fe6c4269ef" containerID="84bd87febf6bcf2b78f3494bd2149a5f8db99063075e70dad47915947f47f62b" exitCode=60
Oct 29 23:00:21.869395 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.869305    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr" event={"ID":"32ea5735-9ca9-4980-8031-44fe6c4269ef","Type":"ContainerDied","Data":"84bd87febf6bcf2b78f3494bd2149a5f8db99063075e70dad47915947f47f62b"}
Oct 29 23:00:21.869828 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.869793    4559 scope.go:122] "RemoveContainer" containerID="84bd87febf6bcf2b78f3494bd2149a5f8db99063075e70dad47915947f47f62b"
Oct 29 23:00:21.870256 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:21.870234    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr_odh-ai-gateway-infra(32ea5735-9ca9-4980-8031-44fe6c4269ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr" podUID="32ea5735-9ca9-4980-8031-44fe6c4269ef"
Oct 29 23:00:21.872208 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.872182    4559 generic.go:356] "Generic (PLEG): container finished" podID="64ae42cd-1e69-44b6-b728-fb34e15e46c7" containerID="81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf" exitCode=60
Oct 29 23:00:21.872639 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.872364    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8" event={"ID":"64ae42cd-1e69-44b6-b728-fb34e15e46c7","Type":"ContainerDied","Data":"81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf"}
Oct 29 23:00:21.873131 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.873114    4559 scope.go:122] "RemoveContainer" containerID="81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf"
Oct 29 23:00:21.874336 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:21.873798    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8_odh-ai-gateway-infra(64ae42cd-1e69-44b6-b728-fb34e15e46c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8" podUID="64ae42cd-1e69-44b6-b728-fb34e15e46c7"
Oct 29 23:00:21.879266 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.879233    4559 generic.go:356] "Generic (PLEG): container finished" podID="40c6ad42-57b2-41c2-a6a5-30fa68a808d4" containerID="31a99fc78e854bb9fe10d98f23f327355c236702bd33e15dee14ee67102516c5" exitCode=60
Oct 29 23:00:21.879423 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.879306    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4" event={"ID":"40c6ad42-57b2-41c2-a6a5-30fa68a808d4","Type":"ContainerDied","Data":"31a99fc78e854bb9fe10d98f23f327355c236702bd33e15dee14ee67102516c5"}
Oct 29 23:00:21.879947 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.879920    4559 scope.go:122] "RemoveContainer" containerID="31a99fc78e854bb9fe10d98f23f327355c236702bd33e15dee14ee67102516c5"
Oct 29 23:00:21.880536 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:21.880498    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4_odh-ai-gateway-infra(40c6ad42-57b2-41c2-a6a5-30fa68a808d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4" podUID="40c6ad42-57b2-41c2-a6a5-30fa68a808d4"
Oct 29 23:00:21.886202 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.886174    4559 generic.go:356] "Generic (PLEG): container finished" podID="3b1c42ea-be0a-4b54-b41c-cd7116493b54" containerID="4ecc722049625680c59e47c545c82c9030f8048e72cbc56d043a6ca902e0cc95" exitCode=60
Oct 29 23:00:21.886342 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.886257    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g" event={"ID":"3b1c42ea-be0a-4b54-b41c-cd7116493b54","Type":"ContainerDied","Data":"4ecc722049625680c59e47c545c82c9030f8048e72cbc56d043a6ca902e0cc95"}
Oct 29 23:00:21.886954 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.886779    4559 scope.go:122] "RemoveContainer" containerID="4ecc722049625680c59e47c545c82c9030f8048e72cbc56d043a6ca902e0cc95"
Oct 29 23:00:21.887298 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:21.887271    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g_odh-ai-gateway-infra(3b1c42ea-be0a-4b54-b41c-cd7116493b54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g" podUID="3b1c42ea-be0a-4b54-b41c-cd7116493b54"
Oct 29 23:00:21.887699 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.887681    4559 scope.go:122] "RemoveContainer" containerID="b870a407077de7f8b5e191591c91014ae270d68524f2f3b55dc6f97308e2d171"
Oct 29 23:00:21.888986 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.888962    4559 generic.go:356] "Generic (PLEG): container finished" podID="7498470d-5bab-4716-8feb-c0e70788cc88" containerID="bf2d4ee10fb8f88a1b2fb37e8e8e9f2f29dd6e5ce9772b238b572b00b65ef1eb" exitCode=60
Oct 29 23:00:21.889102 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.889040    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd" event={"ID":"7498470d-5bab-4716-8feb-c0e70788cc88","Type":"ContainerDied","Data":"bf2d4ee10fb8f88a1b2fb37e8e8e9f2f29dd6e5ce9772b238b572b00b65ef1eb"}
Oct 29 23:00:21.889534 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.889505    4559 scope.go:122] "RemoveContainer" containerID="bf2d4ee10fb8f88a1b2fb37e8e8e9f2f29dd6e5ce9772b238b572b00b65ef1eb"
Oct 29 23:00:21.889929 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:21.889908    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd_odh-ai-gateway-infra(7498470d-5bab-4716-8feb-c0e70788cc88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd" podUID="7498470d-5bab-4716-8feb-c0e70788cc88"
Oct 29 23:00:21.890959 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.890933    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7"]
Oct 29 23:00:21.891393 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.891371    4559 generic.go:356] "Generic (PLEG): container finished" podID="bb6a5323-5dab-421e-9c50-b28825aaa895" containerID="4b978c63699e8ab4ab4ea9f2ed09240271664b8f578990f0dc9414cf32a8e69c" exitCode=60
Oct 29 23:00:21.891481 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.891447    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg" event={"ID":"bb6a5323-5dab-421e-9c50-b28825aaa895","Type":"ContainerDied","Data":"4b978c63699e8ab4ab4ea9f2ed09240271664b8f578990f0dc9414cf32a8e69c"}
Oct 29 23:00:21.891964 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.891932    4559 scope.go:122] "RemoveContainer" containerID="4b978c63699e8ab4ab4ea9f2ed09240271664b8f578990f0dc9414cf32a8e69c"
Oct 29 23:00:21.892270 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:21.892241    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg_odh-ai-gateway-infra(bb6a5323-5dab-421e-9c50-b28825aaa895)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg" podUID="bb6a5323-5dab-421e-9c50-b28825aaa895"
Oct 29 23:00:21.897689 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.897553    4559 generic.go:356] "Generic (PLEG): container finished" podID="624b0508-b13d-41aa-977f-8691e3b0ca01" containerID="c4e77de649c60c71a3675ec5d6c28f4ab83d89f9eb7c3ae43e4c239cebd15a12" exitCode=60
Oct 29 23:00:21.897689 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.897588    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr" event={"ID":"624b0508-b13d-41aa-977f-8691e3b0ca01","Type":"ContainerDied","Data":"c4e77de649c60c71a3675ec5d6c28f4ab83d89f9eb7c3ae43e4c239cebd15a12"}
Oct 29 23:00:21.898244 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.898157    4559 scope.go:122] "RemoveContainer" containerID="c4e77de649c60c71a3675ec5d6c28f4ab83d89f9eb7c3ae43e4c239cebd15a12"
Oct 29 23:00:21.898585 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:21.898562    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr_odh-ai-gateway-infra(624b0508-b13d-41aa-977f-8691e3b0ca01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr" podUID="624b0508-b13d-41aa-977f-8691e3b0ca01"
Oct 29 23:00:21.900166 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.900146    4559 generic.go:356] "Generic (PLEG): container finished" podID="a2e53e77-e3c5-4468-b830-79e768275dde" containerID="e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc" exitCode=60
Oct 29 23:00:21.900262 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.900184    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4" event={"ID":"a2e53e77-e3c5-4468-b830-79e768275dde","Type":"ContainerDied","Data":"e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc"}
Oct 29 23:00:21.900573 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.900554    4559 scope.go:122] "RemoveContainer" containerID="e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc"
Oct 29 23:00:21.900957 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:21.900933    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4_odh-ai-gateway-infra(a2e53e77-e3c5-4468-b830-79e768275dde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4" podUID="a2e53e77-e3c5-4468-b830-79e768275dde"
Oct 29 23:00:21.907090 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.907071    4559 scope.go:122] "RemoveContainer" containerID="74b13009c1a8158dbf96aa1d590aa0e0ab18c45e5c2517b0864b72a3f0987597"
Oct 29 23:00:21.945562 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.945538    4559 scope.go:122] "RemoveContainer" containerID="ba02dc7df266d560eaa4862fc4e7cd73810bc221137f2659c84daadb774131fe"
Oct 29 23:00:21.951409 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.951378    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr"]
Oct 29 23:00:21.983131 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:21.980697    4559 scope.go:122] "RemoveContainer" containerID="da527300b95e84de07189dfc3ad3031f0ff00f186e8a4fa556869878e94322c8"
Oct 29 23:00:22.122751 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:22.122701    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-conmon-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6a5323_5dab_421e_9c50_b28825aaa895.slice/crio-4b978c63699e8ab4ab4ea9f2ed09240271664b8f578990f0dc9414cf32a8e69c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ae42cd_1e69_44b6_b728_fb34e15e46c7.slice/crio-conmon-81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:00:22.128938 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.128909    4559 scope.go:122] "RemoveContainer" containerID="e594e044379c446d9a1b8ea8ad75f9d90ad17a900c1c8eb00bd967e20fa7e061"
Oct 29 23:00:22.147374 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.146591    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg"]
Oct 29 23:00:22.164587 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.164560    4559 scope.go:122] "RemoveContainer" containerID="c55701d20b56244e348e9cd27932c77c08cd7482b93de8a58e127b37480182c7"
Oct 29 23:00:22.169587 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.169562    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd"]
Oct 29 23:00:22.193892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.192331    4559 scope.go:122] "RemoveContainer" containerID="71f34046432cc4a9423c1a5bf4d5cf98c0df47539d5befe9e768ba0516c18b77"
Oct 29 23:00:22.198879 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.198748    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8"]
Oct 29 23:00:22.229899 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.223196    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr"]
Oct 29 23:00:22.229899 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.224504    4559 scope.go:122] "RemoveContainer" containerID="ae7fe318830c58ba690101185728d0f1e68abcad3a794d354d1681895f87066d"
Oct 29 23:00:22.273237 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.272651    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4"]
Oct 29 23:00:22.290596 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.290506    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g"]
Oct 29 23:00:22.295571 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.295546    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt"
Oct 29 23:00:22.312618 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.312591    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4"]
Oct 29 23:00:22.346644 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.345624    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6fdff4a-3510-4774-8c07-dddad4aa7219-kube-api-access-2wfb5\" (UniqueName: \"kubernetes.io/projected/f6fdff4a-3510-4774-8c07-dddad4aa7219-kube-api-access-2wfb5\") pod \"f6fdff4a-3510-4774-8c07-dddad4aa7219\" (UID: \"f6fdff4a-3510-4774-8c07-dddad4aa7219\") "
Oct 29 23:00:22.346644 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.345836    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6fdff4a-3510-4774-8c07-dddad4aa7219-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6fdff4a-3510-4774-8c07-dddad4aa7219-maas-api-service-ca\") pod \"f6fdff4a-3510-4774-8c07-dddad4aa7219\" (UID: \"f6fdff4a-3510-4774-8c07-dddad4aa7219\") "
Oct 29 23:00:22.347138 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.347076    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6fdff4a-3510-4774-8c07-dddad4aa7219-maas-api-service-ca" pod "f6fdff4a-3510-4774-8c07-dddad4aa7219" (UID: "f6fdff4a-3510-4774-8c07-dddad4aa7219"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:22.349302 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.349272    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6fdff4a-3510-4774-8c07-dddad4aa7219-kube-api-access-2wfb5" pod "f6fdff4a-3510-4774-8c07-dddad4aa7219" (UID: "f6fdff4a-3510-4774-8c07-dddad4aa7219"). InnerVolumeSpecName "kube-api-access-2wfb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:22.356517 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.356488    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw"
Oct 29 23:00:22.447733 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.447694    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fb88948-9bd8-4690-bb16-0e4e5cebb1e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb88948-9bd8-4690-bb16-0e4e5cebb1e3-maas-api-service-ca\") pod \"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3\" (UID: \"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3\") "
Oct 29 23:00:22.448098 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.448078    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb88948-9bd8-4690-bb16-0e4e5cebb1e3-kube-api-access-w5kzx\" (UniqueName: \"kubernetes.io/projected/0fb88948-9bd8-4690-bb16-0e4e5cebb1e3-kube-api-access-w5kzx\") pod \"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3\" (UID: \"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3\") "
Oct 29 23:00:22.448570 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.448129    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb88948-9bd8-4690-bb16-0e4e5cebb1e3-maas-api-service-ca" pod "0fb88948-9bd8-4690-bb16-0e4e5cebb1e3" (UID: "0fb88948-9bd8-4690-bb16-0e4e5cebb1e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:22.449314 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.449287    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wfb5\" (UniqueName: \"kubernetes.io/projected/f6fdff4a-3510-4774-8c07-dddad4aa7219-kube-api-access-2wfb5\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.449456 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.449443    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb88948-9bd8-4690-bb16-0e4e5cebb1e3-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.449575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.449564    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6fdff4a-3510-4774-8c07-dddad4aa7219-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.453300 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.453248    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb88948-9bd8-4690-bb16-0e4e5cebb1e3-kube-api-access-w5kzx" pod "0fb88948-9bd8-4690-bb16-0e4e5cebb1e3" (UID: "0fb88948-9bd8-4690-bb16-0e4e5cebb1e3"). InnerVolumeSpecName "kube-api-access-w5kzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:22.551608 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.551559    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5kzx\" (UniqueName: \"kubernetes.io/projected/0fb88948-9bd8-4690-bb16-0e4e5cebb1e3-kube-api-access-w5kzx\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.916585 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.916556    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt"
Oct 29 23:00:22.917021 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.916612    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt" event={"ID":"f6fdff4a-3510-4774-8c07-dddad4aa7219","Type":"ContainerDied","Data":"f1f9690820c6252d366d6f783b00df51a9db45497476b6c42cc45922fa3bdcdf"}
Oct 29 23:00:22.917021 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.916717    4559 scope.go:122] "RemoveContainer" containerID="cc674d0bc7e03d51a3e48566553c58023e058f3e9895c0036278f2d5c1b257d4"
Oct 29 23:00:22.936063 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.935923    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw" event={"ID":"0fb88948-9bd8-4690-bb16-0e4e5cebb1e3","Type":"ContainerDied","Data":"c6da5f48993cd16dd8607bcf937f6409ae9c865e7a67bdbe8c4c5caa521e60d1"}
Oct 29 23:00:22.936063 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.935982    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw"
Oct 29 23:00:22.936063 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:22.935997    4559 scope.go:122] "RemoveContainer" containerID="9ba7e3a52bff2d7f98ee857308d09e441e7dde27536bc27c1c593bd45310aa91"
Oct 29 23:00:23.353988 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.340006    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z" podUID="11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24" containerName="payload-processing" containerID="cri-o://acbc176d1b91b36a2279837f7589acc4c0d3c18de463405ac1730b4c5deb7a99" gracePeriod=25
Oct 29 23:00:23.362694 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.362313    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg" podUID="2b2f4588-6420-4161-a8f1-f7a888ff1993" containerName="payload-pre-processing" containerID="cri-o://a3f8df9d925f6459ed1d048e28ee3603a78a1727d6509bd2eb98cd46806fd44a" gracePeriod=25
Oct 29 23:00:23.451951 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.451918    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:00:23.453501 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.453467    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt"]
Oct 29 23:00:23.466489 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.466409    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-bpvzt"]
Oct 29 23:00:23.611873 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.611658    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4"
Oct 29 23:00:23.644697 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.644657    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c6ad42-57b2-41c2-a6a5-30fa68a808d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c6ad42-57b2-41c2-a6a5-30fa68a808d4-maas-api-service-ca\") pod \"40c6ad42-57b2-41c2-a6a5-30fa68a808d4\" (UID: \"40c6ad42-57b2-41c2-a6a5-30fa68a808d4\") "
Oct 29 23:00:23.645565 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.645540    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c6ad42-57b2-41c2-a6a5-30fa68a808d4-kube-api-access-tv6pq\" (UniqueName: \"kubernetes.io/projected/40c6ad42-57b2-41c2-a6a5-30fa68a808d4-kube-api-access-tv6pq\") pod \"40c6ad42-57b2-41c2-a6a5-30fa68a808d4\" (UID: \"40c6ad42-57b2-41c2-a6a5-30fa68a808d4\") "
Oct 29 23:00:23.646319 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.645270    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c6ad42-57b2-41c2-a6a5-30fa68a808d4-maas-api-service-ca" pod "40c6ad42-57b2-41c2-a6a5-30fa68a808d4" (UID: "40c6ad42-57b2-41c2-a6a5-30fa68a808d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:23.647237 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.647216    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c6ad42-57b2-41c2-a6a5-30fa68a808d4-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:23.652333 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.652281    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c6ad42-57b2-41c2-a6a5-30fa68a808d4-kube-api-access-tv6pq" pod "40c6ad42-57b2-41c2-a6a5-30fa68a808d4" (UID: "40c6ad42-57b2-41c2-a6a5-30fa68a808d4"). InnerVolumeSpecName "kube-api-access-tv6pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:23.725892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.720295    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw"]
Oct 29 23:00:23.789262 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.762911    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv6pq\" (UniqueName: \"kubernetes.io/projected/40c6ad42-57b2-41c2-a6a5-30fa68a808d4-kube-api-access-tv6pq\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:23.789262 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.765464    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d68c2c-29888580-gc2xw"]
Oct 29 23:00:23.971985 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.971953    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd"
Oct 29 23:00:23.979153 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.978646    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg"
Oct 29 23:00:23.985313 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.985237    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8"
Oct 29 23:00:23.994776 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:23.994679    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g"
Oct 29 23:00:24.005286 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.005076    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg" event={"ID":"bb6a5323-5dab-421e-9c50-b28825aaa895","Type":"ContainerDied","Data":"d7194d6cf5ba1f579b9795cf8f4034500ef4ff38c73212453c956d55565cbc82"}
Oct 29 23:00:24.005286 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.005147    4559 scope.go:122] "RemoveContainer" containerID="4b978c63699e8ab4ab4ea9f2ed09240271664b8f578990f0dc9414cf32a8e69c"
Oct 29 23:00:24.005286 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.005160    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg"
Oct 29 23:00:24.005286 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.005084    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4"
Oct 29 23:00:24.010996 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.010919    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr" event={"ID":"624b0508-b13d-41aa-977f-8691e3b0ca01","Type":"ContainerDied","Data":"f262a07aebde4a9778832616738d7eb0dc20b6661978343bfc7e569a82c5c003"}
Oct 29 23:00:24.010996 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.010981    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f262a07aebde4a9778832616738d7eb0dc20b6661978343bfc7e569a82c5c003"
Oct 29 23:00:24.015104 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.014435    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr"
Oct 29 23:00:24.019934 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.017211    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4" event={"ID":"a2e53e77-e3c5-4468-b830-79e768275dde","Type":"ContainerDied","Data":"f43c4188e8c5830f8c57b6f84983c8f5e23963357644e3a66142bcca88b4ae31"}
Oct 29 23:00:24.019934 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.017406    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4"
Oct 29 23:00:24.030070 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.029977    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr"
Oct 29 23:00:24.030736 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.030706    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7" event={"ID":"58b25906-470a-4dbd-b43a-0f0bcd25223c","Type":"ContainerDied","Data":"bca8dacb55ac83d208bad8cf12714844ca3b6a8400ec5969a199ff3f926711e2"}
Oct 29 23:00:24.030901 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.030766    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bca8dacb55ac83d208bad8cf12714844ca3b6a8400ec5969a199ff3f926711e2"
Oct 29 23:00:24.032406 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.032329    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7"
Oct 29 23:00:24.034869 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.032872    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr"
Oct 29 23:00:24.034869 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.033070    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr" event={"ID":"32ea5735-9ca9-4980-8031-44fe6c4269ef","Type":"ContainerDied","Data":"3a6b4fc5b7bfe89b6918e4eb70886bef1b50defd50395213527f8581405ce994"}
Oct 29 23:00:24.050982 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.050927    4559 scope.go:122] "RemoveContainer" containerID="e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc"
Oct 29 23:00:24.122540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.120729    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ea5735-9ca9-4980-8031-44fe6c4269ef-kube-api-access-ml76t\" (UniqueName: \"kubernetes.io/projected/32ea5735-9ca9-4980-8031-44fe6c4269ef-kube-api-access-ml76t\") pod \"32ea5735-9ca9-4980-8031-44fe6c4269ef\" (UID: \"32ea5735-9ca9-4980-8031-44fe6c4269ef\") "
Oct 29 23:00:24.122540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.120852    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e53e77-e3c5-4468-b830-79e768275dde-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e53e77-e3c5-4468-b830-79e768275dde-maas-api-service-ca\") pod \"a2e53e77-e3c5-4468-b830-79e768275dde\" (UID: \"a2e53e77-e3c5-4468-b830-79e768275dde\") "
Oct 29 23:00:24.122540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.120915    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7498470d-5bab-4716-8feb-c0e70788cc88-kube-api-access-c2dcm\" (UniqueName: \"kubernetes.io/projected/7498470d-5bab-4716-8feb-c0e70788cc88-kube-api-access-c2dcm\") pod \"7498470d-5bab-4716-8feb-c0e70788cc88\" (UID: \"7498470d-5bab-4716-8feb-c0e70788cc88\") "
Oct 29 23:00:24.122540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.120980    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ae42cd-1e69-44b6-b728-fb34e15e46c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ae42cd-1e69-44b6-b728-fb34e15e46c7-maas-api-service-ca\") pod \"64ae42cd-1e69-44b6-b728-fb34e15e46c7\" (UID: \"64ae42cd-1e69-44b6-b728-fb34e15e46c7\") "
Oct 29 23:00:24.122540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.121009    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e53e77-e3c5-4468-b830-79e768275dde-kube-api-access-2q2tn\" (UniqueName: \"kubernetes.io/projected/a2e53e77-e3c5-4468-b830-79e768275dde-kube-api-access-2q2tn\") pod \"a2e53e77-e3c5-4468-b830-79e768275dde\" (UID: \"a2e53e77-e3c5-4468-b830-79e768275dde\") "
Oct 29 23:00:24.122540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.121119    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b1c42ea-be0a-4b54-b41c-cd7116493b54-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c42ea-be0a-4b54-b41c-cd7116493b54-maas-api-service-ca\") pod \"3b1c42ea-be0a-4b54-b41c-cd7116493b54\" (UID: \"3b1c42ea-be0a-4b54-b41c-cd7116493b54\") "
Oct 29 23:00:24.122540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.121220    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ea5735-9ca9-4980-8031-44fe6c4269ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ea5735-9ca9-4980-8031-44fe6c4269ef-maas-api-service-ca\") pod \"32ea5735-9ca9-4980-8031-44fe6c4269ef\" (UID: \"32ea5735-9ca9-4980-8031-44fe6c4269ef\") "
Oct 29 23:00:24.122540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.121275    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/624b0508-b13d-41aa-977f-8691e3b0ca01-kube-api-access-9ts8c\" (UniqueName: \"kubernetes.io/projected/624b0508-b13d-41aa-977f-8691e3b0ca01-kube-api-access-9ts8c\") pod \"624b0508-b13d-41aa-977f-8691e3b0ca01\" (UID: \"624b0508-b13d-41aa-977f-8691e3b0ca01\") "
Oct 29 23:00:24.122540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.121350    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ae42cd-1e69-44b6-b728-fb34e15e46c7-kube-api-access-fqdph\" (UniqueName: \"kubernetes.io/projected/64ae42cd-1e69-44b6-b728-fb34e15e46c7-kube-api-access-fqdph\") pod \"64ae42cd-1e69-44b6-b728-fb34e15e46c7\" (UID: \"64ae42cd-1e69-44b6-b728-fb34e15e46c7\") "
Oct 29 23:00:24.122540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.121378    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b1c42ea-be0a-4b54-b41c-cd7116493b54-kube-api-access-g8g2d\" (UniqueName: \"kubernetes.io/projected/3b1c42ea-be0a-4b54-b41c-cd7116493b54-kube-api-access-g8g2d\") pod \"3b1c42ea-be0a-4b54-b41c-cd7116493b54\" (UID: \"3b1c42ea-be0a-4b54-b41c-cd7116493b54\") "
Oct 29 23:00:24.122540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.121437    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58b25906-470a-4dbd-b43a-0f0bcd25223c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b25906-470a-4dbd-b43a-0f0bcd25223c-maas-api-service-ca\") pod \"58b25906-470a-4dbd-b43a-0f0bcd25223c\" (UID: \"58b25906-470a-4dbd-b43a-0f0bcd25223c\") "
Oct 29 23:00:24.122540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.121496    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7498470d-5bab-4716-8feb-c0e70788cc88-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7498470d-5bab-4716-8feb-c0e70788cc88-maas-api-service-ca\") pod \"7498470d-5bab-4716-8feb-c0e70788cc88\" (UID: \"7498470d-5bab-4716-8feb-c0e70788cc88\") "
Oct 29 23:00:24.122540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.121529    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/624b0508-b13d-41aa-977f-8691e3b0ca01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/624b0508-b13d-41aa-977f-8691e3b0ca01-maas-api-service-ca\") pod \"624b0508-b13d-41aa-977f-8691e3b0ca01\" (UID: \"624b0508-b13d-41aa-977f-8691e3b0ca01\") "
Oct 29 23:00:24.122540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.121587    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58b25906-470a-4dbd-b43a-0f0bcd25223c-kube-api-access-mbspv\" (UniqueName: \"kubernetes.io/projected/58b25906-470a-4dbd-b43a-0f0bcd25223c-kube-api-access-mbspv\") pod \"58b25906-470a-4dbd-b43a-0f0bcd25223c\" (UID: \"58b25906-470a-4dbd-b43a-0f0bcd25223c\") "
Oct 29 23:00:24.122540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.121632    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6a5323-5dab-421e-9c50-b28825aaa895-kube-api-access-8w9w4\" (UniqueName: \"kubernetes.io/projected/bb6a5323-5dab-421e-9c50-b28825aaa895-kube-api-access-8w9w4\") pod \"bb6a5323-5dab-421e-9c50-b28825aaa895\" (UID: \"bb6a5323-5dab-421e-9c50-b28825aaa895\") "
Oct 29 23:00:24.122540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.121686    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb6a5323-5dab-421e-9c50-b28825aaa895-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6a5323-5dab-421e-9c50-b28825aaa895-maas-api-service-ca\") pod \"bb6a5323-5dab-421e-9c50-b28825aaa895\" (UID: \"bb6a5323-5dab-421e-9c50-b28825aaa895\") "
Oct 29 23:00:24.123648 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.123092    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6a5323-5dab-421e-9c50-b28825aaa895-maas-api-service-ca" pod "bb6a5323-5dab-421e-9c50-b28825aaa895" (UID: "bb6a5323-5dab-421e-9c50-b28825aaa895"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:24.124932 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.124278    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ae42cd-1e69-44b6-b728-fb34e15e46c7-maas-api-service-ca" pod "64ae42cd-1e69-44b6-b728-fb34e15e46c7" (UID: "64ae42cd-1e69-44b6-b728-fb34e15e46c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:24.124932 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.124624    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e53e77-e3c5-4468-b830-79e768275dde-maas-api-service-ca" pod "a2e53e77-e3c5-4468-b830-79e768275dde" (UID: "a2e53e77-e3c5-4468-b830-79e768275dde"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:24.131797 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.130479    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7498470d-5bab-4716-8feb-c0e70788cc88-maas-api-service-ca" pod "7498470d-5bab-4716-8feb-c0e70788cc88" (UID: "7498470d-5bab-4716-8feb-c0e70788cc88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:24.131797 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.131728    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/624b0508-b13d-41aa-977f-8691e3b0ca01-maas-api-service-ca" pod "624b0508-b13d-41aa-977f-8691e3b0ca01" (UID: "624b0508-b13d-41aa-977f-8691e3b0ca01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:24.132611 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.132585    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624b0508-b13d-41aa-977f-8691e3b0ca01-kube-api-access-9ts8c" pod "624b0508-b13d-41aa-977f-8691e3b0ca01" (UID: "624b0508-b13d-41aa-977f-8691e3b0ca01"). InnerVolumeSpecName "kube-api-access-9ts8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:24.135368 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.133357    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ae42cd-1e69-44b6-b728-fb34e15e46c7-kube-api-access-fqdph" pod "64ae42cd-1e69-44b6-b728-fb34e15e46c7" (UID: "64ae42cd-1e69-44b6-b728-fb34e15e46c7"). InnerVolumeSpecName "kube-api-access-fqdph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:24.135368 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.134360    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1c42ea-be0a-4b54-b41c-cd7116493b54-maas-api-service-ca" pod "3b1c42ea-be0a-4b54-b41c-cd7116493b54" (UID: "3b1c42ea-be0a-4b54-b41c-cd7116493b54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:24.135368 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.134531    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ea5735-9ca9-4980-8031-44fe6c4269ef-kube-api-access-ml76t" pod "32ea5735-9ca9-4980-8031-44fe6c4269ef" (UID: "32ea5735-9ca9-4980-8031-44fe6c4269ef"). InnerVolumeSpecName "kube-api-access-ml76t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:24.135368 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.134952    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ea5735-9ca9-4980-8031-44fe6c4269ef-maas-api-service-ca" pod "32ea5735-9ca9-4980-8031-44fe6c4269ef" (UID: "32ea5735-9ca9-4980-8031-44fe6c4269ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:24.135368 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.135113    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58b25906-470a-4dbd-b43a-0f0bcd25223c-maas-api-service-ca" pod "58b25906-470a-4dbd-b43a-0f0bcd25223c" (UID: "58b25906-470a-4dbd-b43a-0f0bcd25223c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:24.135765 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.135544    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1c42ea-be0a-4b54-b41c-cd7116493b54-kube-api-access-g8g2d" pod "3b1c42ea-be0a-4b54-b41c-cd7116493b54" (UID: "3b1c42ea-be0a-4b54-b41c-cd7116493b54"). InnerVolumeSpecName "kube-api-access-g8g2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:24.135998 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.135959    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7498470d-5bab-4716-8feb-c0e70788cc88-kube-api-access-c2dcm" pod "7498470d-5bab-4716-8feb-c0e70788cc88" (UID: "7498470d-5bab-4716-8feb-c0e70788cc88"). InnerVolumeSpecName "kube-api-access-c2dcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:24.141459 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.141419    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b25906-470a-4dbd-b43a-0f0bcd25223c-kube-api-access-mbspv" pod "58b25906-470a-4dbd-b43a-0f0bcd25223c" (UID: "58b25906-470a-4dbd-b43a-0f0bcd25223c"). InnerVolumeSpecName "kube-api-access-mbspv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:24.141600 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.141487    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e53e77-e3c5-4468-b830-79e768275dde-kube-api-access-2q2tn" pod "a2e53e77-e3c5-4468-b830-79e768275dde" (UID: "a2e53e77-e3c5-4468-b830-79e768275dde"). InnerVolumeSpecName "kube-api-access-2q2tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:24.143341 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.143305    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6a5323-5dab-421e-9c50-b28825aaa895-kube-api-access-8w9w4" pod "bb6a5323-5dab-421e-9c50-b28825aaa895" (UID: "bb6a5323-5dab-421e-9c50-b28825aaa895"). InnerVolumeSpecName "kube-api-access-8w9w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:24.143900 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.143649    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8"
Oct 29 23:00:24.143900 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.143668    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8" event={"ID":"64ae42cd-1e69-44b6-b728-fb34e15e46c7","Type":"ContainerDied","Data":"596ede21fab01675ac5f73cb3b8407c54f1fc5d55c3c681af765e29ab8c0f25e"}
Oct 29 23:00:24.143900 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.147989    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4" event={"ID":"40c6ad42-57b2-41c2-a6a5-30fa68a808d4","Type":"ContainerDied","Data":"4e2005014fbc9437f04000901cc05e32956d84c93f9293ea0d678eab7594a52a"}
Oct 29 23:00:24.143900 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.148105    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4"
Oct 29 23:00:24.151474 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.151218    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g" event={"ID":"3b1c42ea-be0a-4b54-b41c-cd7116493b54","Type":"ContainerDied","Data":"f71bbbc4f93a3b901e8ae8ecc5068c0b76ebdf28494c16d82560b74c05ea12b2"}
Oct 29 23:00:24.151474 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.151326    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g"
Oct 29 23:00:24.157277 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.156475    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd" event={"ID":"7498470d-5bab-4716-8feb-c0e70788cc88","Type":"ContainerDied","Data":"5053bd63cf8d7f0097775ddd3eda477096fcf52b8ccc4ad90d23e56042336f7d"}
Oct 29 23:00:24.157277 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.156637    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd"
Oct 29 23:00:24.165170 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.165052    4559 scope.go:122] "RemoveContainer" containerID="84bd87febf6bcf2b78f3494bd2149a5f8db99063075e70dad47915947f47f62b"
Oct 29 23:00:24.197093 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.195196    4559 scope.go:122] "RemoveContainer" containerID="81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf"
Oct 29 23:00:24.198669 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:24.198539    4559 kuberuntime_gc.go:392] "Failed to remove container log dead symlink" err="remove /var/log/containers/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8_odh-ai-gateway-infra_cleanup-81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf.log: no such file or directory" path="/var/log/containers/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8_odh-ai-gateway-infra_cleanup-81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf.log"
Oct 29 23:00:24.210641 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.210575    4559 scope.go:122] "RemoveContainer" containerID="31a99fc78e854bb9fe10d98f23f327355c236702bd33e15dee14ee67102516c5"
Oct 29 23:00:24.226551 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.226513    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q2tn\" (UniqueName: \"kubernetes.io/projected/a2e53e77-e3c5-4468-b830-79e768275dde-kube-api-access-2q2tn\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.226551 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.226547    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c42ea-be0a-4b54-b41c-cd7116493b54-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.226551 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.226558    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ea5735-9ca9-4980-8031-44fe6c4269ef-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.226850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.226569    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ts8c\" (UniqueName: \"kubernetes.io/projected/624b0508-b13d-41aa-977f-8691e3b0ca01-kube-api-access-9ts8c\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.226850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.226582    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqdph\" (UniqueName: \"kubernetes.io/projected/64ae42cd-1e69-44b6-b728-fb34e15e46c7-kube-api-access-fqdph\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.226850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.226591    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8g2d\" (UniqueName: \"kubernetes.io/projected/3b1c42ea-be0a-4b54-b41c-cd7116493b54-kube-api-access-g8g2d\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.226850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.226601    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b25906-470a-4dbd-b43a-0f0bcd25223c-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.226850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.226611    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7498470d-5bab-4716-8feb-c0e70788cc88-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.226850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.226622    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/624b0508-b13d-41aa-977f-8691e3b0ca01-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.226850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.226630    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbspv\" (UniqueName: \"kubernetes.io/projected/58b25906-470a-4dbd-b43a-0f0bcd25223c-kube-api-access-mbspv\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.226850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.226641    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w9w4\" (UniqueName: \"kubernetes.io/projected/bb6a5323-5dab-421e-9c50-b28825aaa895-kube-api-access-8w9w4\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.226850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.226650    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6a5323-5dab-421e-9c50-b28825aaa895-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.226850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.226662    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml76t\" (UniqueName: \"kubernetes.io/projected/32ea5735-9ca9-4980-8031-44fe6c4269ef-kube-api-access-ml76t\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.226850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.226672    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e53e77-e3c5-4468-b830-79e768275dde-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.226850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.226680    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2dcm\" (UniqueName: \"kubernetes.io/projected/7498470d-5bab-4716-8feb-c0e70788cc88-kube-api-access-c2dcm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.226850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.226690    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ae42cd-1e69-44b6-b728-fb34e15e46c7-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.248097 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.248072    4559 scope.go:122] "RemoveContainer" containerID="4ecc722049625680c59e47c545c82c9030f8048e72cbc56d043a6ca902e0cc95"
Oct 29 23:00:24.261802 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.261769    4559 scope.go:122] "RemoveContainer" containerID="bf2d4ee10fb8f88a1b2fb37e8e8e9f2f29dd6e5ce9772b238b572b00b65ef1eb"
Oct 29 23:00:24.383265 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.383169    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd"]
Oct 29 23:00:24.390999 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.390962    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-83f120-29888580-n6vjd"]
Oct 29 23:00:24.419863 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.419790    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:00:24.420078 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.419892    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m" podUID="f479373b-2190-46af-9b6d-ad901981a56b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:00:24.425201 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.425175    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8"]
Oct 29 23:00:24.434073 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.434037    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-af260f-29888580-hxsz8"]
Oct 29 23:00:24.509936 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.509893    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4"]
Oct 29 23:00:24.512779 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.512701    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888580-rtlf4"]
Oct 29 23:00:24.543788 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.543752    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g"]
Oct 29 23:00:24.548495 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.548461    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888580-h592g"]
Oct 29 23:00:24.581393 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.581360    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4"]
Oct 29 23:00:24.585737 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.585711    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888580-4ztr4"]
Oct 29 23:00:24.622322 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.622287    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr"]
Oct 29 23:00:24.627009 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.626978    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-6fc6bd-29888580-6t9lr"]
Oct 29 23:00:24.662124 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.662088    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg"]
Oct 29 23:00:24.666825 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:24.666782    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888580-9mqlg"]
Oct 29 23:00:25.172048 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.172019    4559 generic.go:356] "Generic (PLEG): container finished" podID="f479373b-2190-46af-9b6d-ad901981a56b" containerID="721665c9e550ef6adbdbf3f4bab54654b871141ed3e1b22105189711c92247c7" exitCode=0
Oct 29 23:00:25.172537 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.172089    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m" event={"ID":"f479373b-2190-46af-9b6d-ad901981a56b","Type":"ContainerDied","Data":"721665c9e550ef6adbdbf3f4bab54654b871141ed3e1b22105189711c92247c7"}
Oct 29 23:00:25.172537 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.172152    4559 scope.go:122] "RemoveContainer" containerID="24dfc707efe15a6ea6c7fb90c9a6b6461ceccf29b006803f913d700c006b1cf7"
Oct 29 23:00:25.177547 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.177521    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr"
Oct 29 23:00:25.177780 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.177532    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7"
Oct 29 23:00:25.202318 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.202296    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 23:00:25.234858 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.234793    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fb88948-9bd8-4690-bb16-0e4e5cebb1e3" path="/var/lib/kubelet/pods/0fb88948-9bd8-4690-bb16-0e4e5cebb1e3/volumes"
Oct 29 23:00:25.235618 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.235587    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ea5735-9ca9-4980-8031-44fe6c4269ef" path="/var/lib/kubelet/pods/32ea5735-9ca9-4980-8031-44fe6c4269ef/volumes"
Oct 29 23:00:25.236317 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.236292    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b1c42ea-be0a-4b54-b41c-cd7116493b54" path="/var/lib/kubelet/pods/3b1c42ea-be0a-4b54-b41c-cd7116493b54/volumes"
Oct 29 23:00:25.237456 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.237422    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c6ad42-57b2-41c2-a6a5-30fa68a808d4" path="/var/lib/kubelet/pods/40c6ad42-57b2-41c2-a6a5-30fa68a808d4/volumes"
Oct 29 23:00:25.238036 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.238017    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ae42cd-1e69-44b6-b728-fb34e15e46c7" path="/var/lib/kubelet/pods/64ae42cd-1e69-44b6-b728-fb34e15e46c7/volumes"
Oct 29 23:00:25.238592 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.238570    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7498470d-5bab-4716-8feb-c0e70788cc88" path="/var/lib/kubelet/pods/7498470d-5bab-4716-8feb-c0e70788cc88/volumes"
Oct 29 23:00:25.239540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.239509    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e53e77-e3c5-4468-b830-79e768275dde" path="/var/lib/kubelet/pods/a2e53e77-e3c5-4468-b830-79e768275dde/volumes"
Oct 29 23:00:25.240147 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.240126    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb6a5323-5dab-421e-9c50-b28825aaa895" path="/var/lib/kubelet/pods/bb6a5323-5dab-421e-9c50-b28825aaa895/volumes"
Oct 29 23:00:25.240668 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.240650    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6fdff4a-3510-4774-8c07-dddad4aa7219" path="/var/lib/kubelet/pods/f6fdff4a-3510-4774-8c07-dddad4aa7219/volumes"
Oct 29 23:00:25.247851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.247795    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7"]
Oct 29 23:00:25.253472 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.253435    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888580-z6pz7"]
Oct 29 23:00:25.287470 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.287431    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr"]
Oct 29 23:00:25.292758 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.292727    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7be71d-29888580-4ltvr"]
Oct 29 23:00:25.367087 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.367052    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-credential-socket\") pod \"f479373b-2190-46af-9b6d-ad901981a56b\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") "
Oct 29 23:00:25.367278 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.367100    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-workload-socket\") pod \"f479373b-2190-46af-9b6d-ad901981a56b\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") "
Oct 29 23:00:25.367278 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.367139    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-workload-certs\") pod \"f479373b-2190-46af-9b6d-ad901981a56b\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") "
Oct 29 23:00:25.367278 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.367188    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-istio-envoy\") pod \"f479373b-2190-46af-9b6d-ad901981a56b\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") "
Oct 29 23:00:25.367505 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.367475    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-workload-certs" pod "f479373b-2190-46af-9b6d-ad901981a56b" (UID: "f479373b-2190-46af-9b6d-ad901981a56b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:25.367583 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.367475    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-credential-socket" pod "f479373b-2190-46af-9b6d-ad901981a56b" (UID: "f479373b-2190-46af-9b6d-ad901981a56b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:25.367583 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.367525    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f479373b-2190-46af-9b6d-ad901981a56b-kube-api-access-hj4bs\" (UniqueName: \"kubernetes.io/projected/f479373b-2190-46af-9b6d-ad901981a56b-kube-api-access-hj4bs\") pod \"f479373b-2190-46af-9b6d-ad901981a56b\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") "
Oct 29 23:00:25.367689 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.367578    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f479373b-2190-46af-9b6d-ad901981a56b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f479373b-2190-46af-9b6d-ad901981a56b-istio-podinfo\") pod \"f479373b-2190-46af-9b6d-ad901981a56b\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") "
Oct 29 23:00:25.367743 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.367719    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f479373b-2190-46af-9b6d-ad901981a56b-istio-token\" (UniqueName: \"kubernetes.io/projected/f479373b-2190-46af-9b6d-ad901981a56b-istio-token\") pod \"f479373b-2190-46af-9b6d-ad901981a56b\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") "
Oct 29 23:00:25.367798 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.367767    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f479373b-2190-46af-9b6d-ad901981a56b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f479373b-2190-46af-9b6d-ad901981a56b-istiod-ca-cert\") pod \"f479373b-2190-46af-9b6d-ad901981a56b\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") "
Oct 29 23:00:25.367874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.367802    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-istio-data\") pod \"f479373b-2190-46af-9b6d-ad901981a56b\" (UID: \"f479373b-2190-46af-9b6d-ad901981a56b\") "
Oct 29 23:00:25.368747 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.368714    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-workload-socket" pod "f479373b-2190-46af-9b6d-ad901981a56b" (UID: "f479373b-2190-46af-9b6d-ad901981a56b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:25.369952 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.368972    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-istio-data" pod "f479373b-2190-46af-9b6d-ad901981a56b" (UID: "f479373b-2190-46af-9b6d-ad901981a56b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:25.369952 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.368989    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.369952 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.369013    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.369952 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.369029    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.370427 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.370395    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f479373b-2190-46af-9b6d-ad901981a56b-istiod-ca-cert" pod "f479373b-2190-46af-9b6d-ad901981a56b" (UID: "f479373b-2190-46af-9b6d-ad901981a56b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:25.370579 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.370552    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-istio-envoy" pod "f479373b-2190-46af-9b6d-ad901981a56b" (UID: "f479373b-2190-46af-9b6d-ad901981a56b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:25.370579 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.370567    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f479373b-2190-46af-9b6d-ad901981a56b-istio-token" pod "f479373b-2190-46af-9b6d-ad901981a56b" (UID: "f479373b-2190-46af-9b6d-ad901981a56b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:25.370700 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.370601    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f479373b-2190-46af-9b6d-ad901981a56b-kube-api-access-hj4bs" pod "f479373b-2190-46af-9b6d-ad901981a56b" (UID: "f479373b-2190-46af-9b6d-ad901981a56b"). InnerVolumeSpecName "kube-api-access-hj4bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:25.370995 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.370967    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f479373b-2190-46af-9b6d-ad901981a56b-istio-podinfo" pod "f479373b-2190-46af-9b6d-ad901981a56b" (UID: "f479373b-2190-46af-9b6d-ad901981a56b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:25.471218 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.471129    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.471218 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.471169    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f479373b-2190-46af-9b6d-ad901981a56b-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.471218 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.471183    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj4bs\" (UniqueName: \"kubernetes.io/projected/f479373b-2190-46af-9b6d-ad901981a56b-kube-api-access-hj4bs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.471218 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.471202    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f479373b-2190-46af-9b6d-ad901981a56b-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.471218 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.471216    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f479373b-2190-46af-9b6d-ad901981a56b-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.472333 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:25.471228    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f479373b-2190-46af-9b6d-ad901981a56b-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:26.087114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:26.087067    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg" podUID="2b2f4588-6420-4161-a8f1-f7a888ff1993" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:26.087416 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:26.087067    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z" podUID="11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:26.187477 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:26.187451    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"
Oct 29 23:00:26.188014 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:26.187492    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m" event={"ID":"f479373b-2190-46af-9b6d-ad901981a56b","Type":"ContainerDied","Data":"ffe798ab6b4b31a95f4ca07156ebd00c723bb15406c92c2e16b47dbb05738ceb"}
Oct 29 23:00:26.188014 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:26.187559    4559 scope.go:122] "RemoveContainer" containerID="721665c9e550ef6adbdbf3f4bab54654b871141ed3e1b22105189711c92247c7"
Oct 29 23:00:26.306838 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:26.306782    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"]
Oct 29 23:00:26.311496 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:26.311465    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-qmc4m"]
Oct 29 23:00:27.229785 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:27.229748    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58b25906-470a-4dbd-b43a-0f0bcd25223c" path="/var/lib/kubelet/pods/58b25906-470a-4dbd-b43a-0f0bcd25223c/volumes"
Oct 29 23:00:27.230275 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:27.230259    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="624b0508-b13d-41aa-977f-8691e3b0ca01" path="/var/lib/kubelet/pods/624b0508-b13d-41aa-977f-8691e3b0ca01/volumes"
Oct 29 23:00:27.230733 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:27.230719    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f479373b-2190-46af-9b6d-ad901981a56b" path="/var/lib/kubelet/pods/f479373b-2190-46af-9b6d-ad901981a56b/volumes"
Oct 29 23:00:30.526255 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:30.526213    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"]
Oct 29 23:00:30.526769 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:30.526429    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958" podUID="1423563c-a109-46a3-9fd5-cccab25939d3" containerName="istio-proxy" containerID="cri-o://23184661f506410278cae1b4a7f9a1073c924b89f83535a864bf92a80157285e" gracePeriod=30
Oct 29 23:00:31.578873 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:31.572136    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ae42cd_1e69_44b6_b728_fb34e15e46c7.slice/crio-conmon-81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-conmon-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6a5323_5dab_421e_9c50_b28825aaa895.slice/crio-4b978c63699e8ab4ab4ea9f2ed09240271664b8f578990f0dc9414cf32a8e69c.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:00:33.920297 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:33.919057    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw_d5789ad1-f64c-4eb8-9a38-49da59afb179/payload-processing/0.log"
Oct 29 23:00:34.105145 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.105072    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw_d5789ad1-f64c-4eb8-9a38-49da59afb179/payload-processing/0.log"
Oct 29 23:00:34.616393 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.616341    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"]
Oct 29 23:00:34.651658 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.651615    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"]
Oct 29 23:00:34.934209 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.934174    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"]
Oct 29 23:00:34.935800 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.935777    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bb6a5323-5dab-421e-9c50-b28825aaa895" containerName="cleanup"
Oct 29 23:00:34.935887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.935804    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6a5323-5dab-421e-9c50-b28825aaa895" containerName="cleanup"
Oct 29 23:00:34.935948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.935904    4559 cpu_manager.go:414] "containerMap: removing container" podUID="f6fdff4a-3510-4774-8c07-dddad4aa7219" containerName="cleanup"
Oct 29 23:00:34.935948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.935914    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6fdff4a-3510-4774-8c07-dddad4aa7219" containerName="cleanup"
Oct 29 23:00:34.935948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.935924    4559 cpu_manager.go:414] "containerMap: removing container" podUID="58b25906-470a-4dbd-b43a-0f0bcd25223c" containerName="cleanup"
Oct 29 23:00:34.935948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.935932    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58b25906-470a-4dbd-b43a-0f0bcd25223c" containerName="cleanup"
Oct 29 23:00:34.936077 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.935950    4559 cpu_manager.go:414] "containerMap: removing container" podUID="f479373b-2190-46af-9b6d-ad901981a56b" containerName="istio-proxy"
Oct 29 23:00:34.936077 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.935957    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f479373b-2190-46af-9b6d-ad901981a56b" containerName="istio-proxy"
Oct 29 23:00:34.936077 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.935968    4559 cpu_manager.go:414] "containerMap: removing container" podUID="32ea5735-9ca9-4980-8031-44fe6c4269ef" containerName="cleanup"
Oct 29 23:00:34.936077 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.935975    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ea5735-9ca9-4980-8031-44fe6c4269ef" containerName="cleanup"
Oct 29 23:00:34.936077 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.935988    4559 cpu_manager.go:414] "containerMap: removing container" podUID="fed652a5-91ae-4f75-975a-2d858e4c3bb9" containerName="istio-proxy"
Oct 29 23:00:34.936077 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.935995    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed652a5-91ae-4f75-975a-2d858e4c3bb9" containerName="istio-proxy"
Oct 29 23:00:34.936077 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936005    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7498470d-5bab-4716-8feb-c0e70788cc88" containerName="cleanup"
Oct 29 23:00:34.936077 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936013    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7498470d-5bab-4716-8feb-c0e70788cc88" containerName="cleanup"
Oct 29 23:00:34.936077 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936026    4559 cpu_manager.go:414] "containerMap: removing container" podUID="58b25906-470a-4dbd-b43a-0f0bcd25223c" containerName="cleanup"
Oct 29 23:00:34.936077 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936034    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58b25906-470a-4dbd-b43a-0f0bcd25223c" containerName="cleanup"
Oct 29 23:00:34.936077 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936045    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3b1c42ea-be0a-4b54-b41c-cd7116493b54" containerName="cleanup"
Oct 29 23:00:34.936077 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936052    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1c42ea-be0a-4b54-b41c-cd7116493b54" containerName="cleanup"
Oct 29 23:00:34.936077 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936066    4559 cpu_manager.go:414] "containerMap: removing container" podUID="624b0508-b13d-41aa-977f-8691e3b0ca01" containerName="cleanup"
Oct 29 23:00:34.936077 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936073    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="624b0508-b13d-41aa-977f-8691e3b0ca01" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936099    4559 cpu_manager.go:414] "containerMap: removing container" podUID="32ea5735-9ca9-4980-8031-44fe6c4269ef" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936106    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ea5735-9ca9-4980-8031-44fe6c4269ef" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936119    4559 cpu_manager.go:414] "containerMap: removing container" podUID="32ea5735-9ca9-4980-8031-44fe6c4269ef" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936126    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ea5735-9ca9-4980-8031-44fe6c4269ef" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936134    4559 cpu_manager.go:414] "containerMap: removing container" podUID="94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5" containerName="revoke-keys"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936141    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5" containerName="revoke-keys"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936151    4559 cpu_manager.go:414] "containerMap: removing container" podUID="971d539d-ac5b-4684-8897-eb299ae4c2dc" containerName="revoke-keys"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936157    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="971d539d-ac5b-4684-8897-eb299ae4c2dc" containerName="revoke-keys"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936170    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bde5a64f-e410-46b8-8e6f-c2933360770a" containerName="istio-proxy"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936177    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde5a64f-e410-46b8-8e6f-c2933360770a" containerName="istio-proxy"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936187    4559 cpu_manager.go:414] "containerMap: removing container" podUID="619b371d-35a7-43d9-ace8-cfc478f2fdfa" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936194    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619b371d-35a7-43d9-ace8-cfc478f2fdfa" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936208    4559 cpu_manager.go:414] "containerMap: removing container" podUID="f6fdff4a-3510-4774-8c07-dddad4aa7219" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936215    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6fdff4a-3510-4774-8c07-dddad4aa7219" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936229    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0fb88948-9bd8-4690-bb16-0e4e5cebb1e3" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936237    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb88948-9bd8-4690-bb16-0e4e5cebb1e3" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936255    4559 cpu_manager.go:414] "containerMap: removing container" podUID="40c6ad42-57b2-41c2-a6a5-30fa68a808d4" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936261    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c6ad42-57b2-41c2-a6a5-30fa68a808d4" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936270    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7498470d-5bab-4716-8feb-c0e70788cc88" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936277    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7498470d-5bab-4716-8feb-c0e70788cc88" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936288    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0fb88948-9bd8-4690-bb16-0e4e5cebb1e3" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936315    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb88948-9bd8-4690-bb16-0e4e5cebb1e3" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936332    4559 cpu_manager.go:414] "containerMap: removing container" podUID="64ae42cd-1e69-44b6-b728-fb34e15e46c7" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936339    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ae42cd-1e69-44b6-b728-fb34e15e46c7" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936359    4559 cpu_manager.go:414] "containerMap: removing container" podUID="f479373b-2190-46af-9b6d-ad901981a56b" containerName="istio-proxy"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936367    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f479373b-2190-46af-9b6d-ad901981a56b" containerName="istio-proxy"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936380    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bb6a5323-5dab-421e-9c50-b28825aaa895" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936387    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6a5323-5dab-421e-9c50-b28825aaa895" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936401    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a2e53e77-e3c5-4468-b830-79e768275dde" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936408    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e53e77-e3c5-4468-b830-79e768275dde" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936421    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a2e53e77-e3c5-4468-b830-79e768275dde" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936427    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e53e77-e3c5-4468-b830-79e768275dde" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936439    4559 cpu_manager.go:414] "containerMap: removing container" podUID="624b0508-b13d-41aa-977f-8691e3b0ca01" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936446    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="624b0508-b13d-41aa-977f-8691e3b0ca01" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936456    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e824b452-5c7a-4d42-8f86-bf43dd571680" containerName="maas-api"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936463    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e824b452-5c7a-4d42-8f86-bf43dd571680" containerName="maas-api"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936476    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0fb88948-9bd8-4690-bb16-0e4e5cebb1e3" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936484    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb88948-9bd8-4690-bb16-0e4e5cebb1e3" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936493    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a2e53e77-e3c5-4468-b830-79e768275dde" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936499    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e53e77-e3c5-4468-b830-79e768275dde" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936514    4559 cpu_manager.go:414] "containerMap: removing container" podUID="64ae42cd-1e69-44b6-b728-fb34e15e46c7" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936521    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ae42cd-1e69-44b6-b728-fb34e15e46c7" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936538    4559 cpu_manager.go:414] "containerMap: removing container" podUID="624b0508-b13d-41aa-977f-8691e3b0ca01" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936544    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="624b0508-b13d-41aa-977f-8691e3b0ca01" containerName="cleanup"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936574    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bde5a64f-e410-46b8-8e6f-c2933360770a" containerName="istio-proxy"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936583    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde5a64f-e410-46b8-8e6f-c2933360770a" containerName="istio-proxy"
Oct 29 23:00:34.936575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936594    4559 cpu_manager.go:414] "containerMap: removing container" podUID="064c3a9c-29ac-48fa-9382-1a8ac20fb18a" containerName="istio-proxy"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936601    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="064c3a9c-29ac-48fa-9382-1a8ac20fb18a" containerName="istio-proxy"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936619    4559 cpu_manager.go:414] "containerMap: removing container" podUID="619b371d-35a7-43d9-ace8-cfc478f2fdfa" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936626    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619b371d-35a7-43d9-ace8-cfc478f2fdfa" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936638    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3b1c42ea-be0a-4b54-b41c-cd7116493b54" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936645    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1c42ea-be0a-4b54-b41c-cd7116493b54" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936657    4559 cpu_manager.go:414] "containerMap: removing container" podUID="40c6ad42-57b2-41c2-a6a5-30fa68a808d4" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936665    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c6ad42-57b2-41c2-a6a5-30fa68a808d4" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936681    4559 cpu_manager.go:414] "containerMap: removing container" podUID="84e86523-8b8b-446d-a5de-25172d07da2d" containerName="istio-proxy"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936687    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e86523-8b8b-446d-a5de-25172d07da2d" containerName="istio-proxy"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936698    4559 cpu_manager.go:414] "containerMap: removing container" podUID="64ae42cd-1e69-44b6-b728-fb34e15e46c7" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.936706    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ae42cd-1e69-44b6-b728-fb34e15e46c7" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937014    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="58b25906-470a-4dbd-b43a-0f0bcd25223c" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937035    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1c42ea-be0a-4b54-b41c-cd7116493b54" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937047    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="624b0508-b13d-41aa-977f-8691e3b0ca01" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937062    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ea5735-9ca9-4980-8031-44fe6c4269ef" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937070    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ea5735-9ca9-4980-8031-44fe6c4269ef" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937080    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5" containerName="revoke-keys"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937090    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="971d539d-ac5b-4684-8897-eb299ae4c2dc" containerName="revoke-keys"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937101    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde5a64f-e410-46b8-8e6f-c2933360770a" containerName="istio-proxy"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937110    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="619b371d-35a7-43d9-ace8-cfc478f2fdfa" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937119    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6fdff4a-3510-4774-8c07-dddad4aa7219" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937129    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb88948-9bd8-4690-bb16-0e4e5cebb1e3" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937139    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c6ad42-57b2-41c2-a6a5-30fa68a808d4" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937148    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7498470d-5bab-4716-8feb-c0e70788cc88" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937158    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb88948-9bd8-4690-bb16-0e4e5cebb1e3" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937168    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ae42cd-1e69-44b6-b728-fb34e15e46c7" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937177    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="f479373b-2190-46af-9b6d-ad901981a56b" containerName="istio-proxy"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937185    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6a5323-5dab-421e-9c50-b28825aaa895" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937194    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e53e77-e3c5-4468-b830-79e768275dde" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937205    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="619b371d-35a7-43d9-ace8-cfc478f2fdfa" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937212    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e53e77-e3c5-4468-b830-79e768275dde" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937224    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="624b0508-b13d-41aa-977f-8691e3b0ca01" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937233    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e824b452-5c7a-4d42-8f86-bf43dd571680" containerName="maas-api"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937244    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb88948-9bd8-4690-bb16-0e4e5cebb1e3" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937252    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e53e77-e3c5-4468-b830-79e768275dde" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937260    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ae42cd-1e69-44b6-b728-fb34e15e46c7" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937270    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="624b0508-b13d-41aa-977f-8691e3b0ca01" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937286    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="064c3a9c-29ac-48fa-9382-1a8ac20fb18a" containerName="istio-proxy"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937297    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1c42ea-be0a-4b54-b41c-cd7116493b54" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937305    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6a5323-5dab-421e-9c50-b28825aaa895" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937313    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c6ad42-57b2-41c2-a6a5-30fa68a808d4" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937323    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e86523-8b8b-446d-a5de-25172d07da2d" containerName="istio-proxy"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937333    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ae42cd-1e69-44b6-b728-fb34e15e46c7" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937343    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6a5323-5dab-421e-9c50-b28825aaa895" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937351    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6fdff4a-3510-4774-8c07-dddad4aa7219" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937360    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="58b25906-470a-4dbd-b43a-0f0bcd25223c" containerName="cleanup"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937371    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="f479373b-2190-46af-9b6d-ad901981a56b" containerName="istio-proxy"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937379    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed652a5-91ae-4f75-975a-2d858e4c3bb9" containerName="istio-proxy"
Oct 29 23:00:34.939180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.937388    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7498470d-5bab-4716-8feb-c0e70788cc88" containerName="cleanup"
Oct 29 23:00:34.950267 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.950237    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"
Oct 29 23:00:34.950616 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.950589    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"]
Oct 29 23:00:34.966028 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.965954    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:00:34.966028 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:34.965998    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:00:35.046635 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.046558    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/199dffea-70ca-4137-90f7-9349eeac1660-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg\" (UID: \"199dffea-70ca-4137-90f7-9349eeac1660\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"
Oct 29 23:00:35.048213 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.048171    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtcxl\" (UniqueName: \"kubernetes.io/projected/199dffea-70ca-4137-90f7-9349eeac1660-kube-api-access-rtcxl\") pod \"maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg\" (UID: \"199dffea-70ca-4137-90f7-9349eeac1660\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"
Oct 29 23:00:35.048400 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.048265    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/199dffea-70ca-4137-90f7-9349eeac1660-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg\" (UID: \"199dffea-70ca-4137-90f7-9349eeac1660\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"
Oct 29 23:00:35.149878 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.149835    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/199dffea-70ca-4137-90f7-9349eeac1660-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg\" (UID: \"199dffea-70ca-4137-90f7-9349eeac1660\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"
Oct 29 23:00:35.150102 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.149911    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtcxl\" (UniqueName: \"kubernetes.io/projected/199dffea-70ca-4137-90f7-9349eeac1660-kube-api-access-rtcxl\") pod \"maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg\" (UID: \"199dffea-70ca-4137-90f7-9349eeac1660\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"
Oct 29 23:00:35.150102 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.149945    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/199dffea-70ca-4137-90f7-9349eeac1660-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg\" (UID: \"199dffea-70ca-4137-90f7-9349eeac1660\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"
Oct 29 23:00:35.150448 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.150424    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/199dffea-70ca-4137-90f7-9349eeac1660-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg\" (UID: \"199dffea-70ca-4137-90f7-9349eeac1660\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"
Oct 29 23:00:35.152599 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.152573    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/199dffea-70ca-4137-90f7-9349eeac1660-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg\" (UID: \"199dffea-70ca-4137-90f7-9349eeac1660\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"
Oct 29 23:00:35.169893 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.169856    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtcxl\" (UniqueName: \"kubernetes.io/projected/199dffea-70ca-4137-90f7-9349eeac1660-kube-api-access-rtcxl\") pod \"maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg\" (UID: \"199dffea-70ca-4137-90f7-9349eeac1660\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"
Oct 29 23:00:35.286997 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.282571    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"
Oct 29 23:00:35.850955 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:00:35.850585    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199dffea_70ca_4137_90f7_9349eeac1660.slice/crio-901a8570e36b50953620bc4c9515ac2608f25aa036ff15fead67d93e1dc04718 WatchSource:0}: Error finding container 901a8570e36b50953620bc4c9515ac2608f25aa036ff15fead67d93e1dc04718: Status 404 returned error can't find the container with id 901a8570e36b50953620bc4c9515ac2608f25aa036ff15fead67d93e1dc04718
Oct 29 23:00:35.853224 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.852848    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"]
Oct 29 23:00:35.869059 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.869027    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 23:00:35.996941 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.996905    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1423563c-a109-46a3-9fd5-cccab25939d3-kube-api-access-k4gpp\" (UniqueName: \"kubernetes.io/projected/1423563c-a109-46a3-9fd5-cccab25939d3-kube-api-access-k4gpp\") pod \"1423563c-a109-46a3-9fd5-cccab25939d3\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") "
Oct 29 23:00:35.997419 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.996979    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1423563c-a109-46a3-9fd5-cccab25939d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1423563c-a109-46a3-9fd5-cccab25939d3-istio-podinfo\") pod \"1423563c-a109-46a3-9fd5-cccab25939d3\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") "
Oct 29 23:00:35.997419 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.997058    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-workload-socket\") pod \"1423563c-a109-46a3-9fd5-cccab25939d3\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") "
Oct 29 23:00:35.997419 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.997198    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1423563c-a109-46a3-9fd5-cccab25939d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1423563c-a109-46a3-9fd5-cccab25939d3-istiod-ca-cert\") pod \"1423563c-a109-46a3-9fd5-cccab25939d3\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") "
Oct 29 23:00:35.997419 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.997260    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1423563c-a109-46a3-9fd5-cccab25939d3-istio-token\" (UniqueName: \"kubernetes.io/projected/1423563c-a109-46a3-9fd5-cccab25939d3-istio-token\") pod \"1423563c-a109-46a3-9fd5-cccab25939d3\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") "
Oct 29 23:00:35.997419 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.997292    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-istio-envoy\") pod \"1423563c-a109-46a3-9fd5-cccab25939d3\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") "
Oct 29 23:00:35.997419 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.997391    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-istio-data\") pod \"1423563c-a109-46a3-9fd5-cccab25939d3\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") "
Oct 29 23:00:35.997787 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.997423    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-workload-socket" pod "1423563c-a109-46a3-9fd5-cccab25939d3" (UID: "1423563c-a109-46a3-9fd5-cccab25939d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:35.997787 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.997501    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-workload-certs\") pod \"1423563c-a109-46a3-9fd5-cccab25939d3\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") "
Oct 29 23:00:35.997787 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.997570    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1423563c-a109-46a3-9fd5-cccab25939d3-istiod-ca-cert" pod "1423563c-a109-46a3-9fd5-cccab25939d3" (UID: "1423563c-a109-46a3-9fd5-cccab25939d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:35.997787 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.997577    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-credential-socket\") pod \"1423563c-a109-46a3-9fd5-cccab25939d3\" (UID: \"1423563c-a109-46a3-9fd5-cccab25939d3\") "
Oct 29 23:00:35.997787 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.997745    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-istio-data" pod "1423563c-a109-46a3-9fd5-cccab25939d3" (UID: "1423563c-a109-46a3-9fd5-cccab25939d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:35.998260 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.997787    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-credential-socket" pod "1423563c-a109-46a3-9fd5-cccab25939d3" (UID: "1423563c-a109-46a3-9fd5-cccab25939d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:35.998260 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.997907    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-workload-certs" pod "1423563c-a109-46a3-9fd5-cccab25939d3" (UID: "1423563c-a109-46a3-9fd5-cccab25939d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:35.999948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.998932    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:35.999948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.999184    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:35.999948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.999207    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1423563c-a109-46a3-9fd5-cccab25939d3-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:35.999948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.999220    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:35.999948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:35.999234    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:36.039045 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.038984    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1423563c-a109-46a3-9fd5-cccab25939d3-istio-podinfo" pod "1423563c-a109-46a3-9fd5-cccab25939d3" (UID: "1423563c-a109-46a3-9fd5-cccab25939d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:36.039248 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.039069    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-istio-envoy" pod "1423563c-a109-46a3-9fd5-cccab25939d3" (UID: "1423563c-a109-46a3-9fd5-cccab25939d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:36.040089 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.040061    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1423563c-a109-46a3-9fd5-cccab25939d3-istio-token" pod "1423563c-a109-46a3-9fd5-cccab25939d3" (UID: "1423563c-a109-46a3-9fd5-cccab25939d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:36.040468 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.040437    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1423563c-a109-46a3-9fd5-cccab25939d3-kube-api-access-k4gpp" pod "1423563c-a109-46a3-9fd5-cccab25939d3" (UID: "1423563c-a109-46a3-9fd5-cccab25939d3"). InnerVolumeSpecName "kube-api-access-k4gpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:36.084880 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.084785    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg" podUID="2b2f4588-6420-4161-a8f1-f7a888ff1993" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:36.085106 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.085002    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z" podUID="11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:36.101355 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.101251    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1423563c-a109-46a3-9fd5-cccab25939d3-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:36.101355 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.101291    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1423563c-a109-46a3-9fd5-cccab25939d3-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:36.101355 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.101306    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4gpp\" (UniqueName: \"kubernetes.io/projected/1423563c-a109-46a3-9fd5-cccab25939d3-kube-api-access-k4gpp\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:36.101355 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.101323    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1423563c-a109-46a3-9fd5-cccab25939d3-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:36.526723 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.526682    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg" event={"ID":"199dffea-70ca-4137-90f7-9349eeac1660","Type":"ContainerStarted","Data":"980d3d48d02cbdb6656f0ebb7c6493c10fdc87b7350d63b17b54b4e0b652cb96"}
Oct 29 23:00:36.526966 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.526743    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg" event={"ID":"199dffea-70ca-4137-90f7-9349eeac1660","Type":"ContainerStarted","Data":"901a8570e36b50953620bc4c9515ac2608f25aa036ff15fead67d93e1dc04718"}
Oct 29 23:00:36.528840 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.528787    4559 generic.go:356] "Generic (PLEG): container finished" podID="1423563c-a109-46a3-9fd5-cccab25939d3" containerID="23184661f506410278cae1b4a7f9a1073c924b89f83535a864bf92a80157285e" exitCode=0
Oct 29 23:00:36.528983 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.528849    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"
Oct 29 23:00:36.528983 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.528843    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958" event={"ID":"1423563c-a109-46a3-9fd5-cccab25939d3","Type":"ContainerDied","Data":"23184661f506410278cae1b4a7f9a1073c924b89f83535a864bf92a80157285e"}
Oct 29 23:00:36.529079 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.528992    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958" event={"ID":"1423563c-a109-46a3-9fd5-cccab25939d3","Type":"ContainerDied","Data":"6e5abe777d0639cfba67b459d0eee9e7f743e2dfe1e9ce42d4740644fd4e6f8b"}
Oct 29 23:00:36.529079 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.529032    4559 scope.go:122] "RemoveContainer" containerID="23184661f506410278cae1b4a7f9a1073c924b89f83535a864bf92a80157285e"
Oct 29 23:00:36.545375 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.545342    4559 scope.go:122] "RemoveContainer" containerID="23184661f506410278cae1b4a7f9a1073c924b89f83535a864bf92a80157285e"
Oct 29 23:00:36.545839 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:36.545789    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23184661f506410278cae1b4a7f9a1073c924b89f83535a864bf92a80157285e\": container with ID starting with 23184661f506410278cae1b4a7f9a1073c924b89f83535a864bf92a80157285e not found: ID does not exist" containerID="23184661f506410278cae1b4a7f9a1073c924b89f83535a864bf92a80157285e"
Oct 29 23:00:36.545957 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.545847    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23184661f506410278cae1b4a7f9a1073c924b89f83535a864bf92a80157285e"} err="failed to get container status \"23184661f506410278cae1b4a7f9a1073c924b89f83535a864bf92a80157285e\": rpc error: code = NotFound desc = could not find container \"23184661f506410278cae1b4a7f9a1073c924b89f83535a864bf92a80157285e\": container with ID starting with 23184661f506410278cae1b4a7f9a1073c924b89f83535a864bf92a80157285e not found: ID does not exist"
Oct 29 23:00:36.571736 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.571679    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg" podStartSLOduration=2.5716654869999998 podStartE2EDuration="2.571665487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:36.566680858 +0000 UTC m=+2775.478124416" watchObservedRunningTime="2026-10-29 23:00:36.571665487 +0000 UTC m=+2775.483109086"
Oct 29 23:00:36.606319 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.606284    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"]
Oct 29 23:00:36.618680 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.618641    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-nn958"]
Oct 29 23:00:36.990419 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.990378    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:00:36.990655 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:36.990438    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs" podUID="6f1b07f5-c3bd-4382-9318-2234d4e9093e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:37.175607 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:37.175561    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6a5323_5dab_421e_9c50_b28825aaa895.slice/crio-4b978c63699e8ab4ab4ea9f2ed09240271664b8f578990f0dc9414cf32a8e69c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-conmon-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@32-41685-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1b07f5_c3bd_4382_9318_2234d4e9093e.slice/crio-be840790c395da14e994e5eed7278ff4ab0f9d97d887505937f7ee3cc9f1e3ff.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ae42cd_1e69_44b6_b728_fb34e15e46c7.slice/crio-conmon-81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:00:37.231578 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:37.231541    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1423563c-a109-46a3-9fd5-cccab25939d3" path="/var/lib/kubelet/pods/1423563c-a109-46a3-9fd5-cccab25939d3/volumes"
Oct 29 23:00:37.538790 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:37.538752    4559 generic.go:356] "Generic (PLEG): container finished" podID="199dffea-70ca-4137-90f7-9349eeac1660" containerID="980d3d48d02cbdb6656f0ebb7c6493c10fdc87b7350d63b17b54b4e0b652cb96" exitCode=0
Oct 29 23:00:37.539084 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:37.538874    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg" event={"ID":"199dffea-70ca-4137-90f7-9349eeac1660","Type":"ContainerDied","Data":"980d3d48d02cbdb6656f0ebb7c6493c10fdc87b7350d63b17b54b4e0b652cb96"}
Oct 29 23:00:37.540907 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:37.540880    4559 generic.go:356] "Generic (PLEG): container finished" podID="6f1b07f5-c3bd-4382-9318-2234d4e9093e" containerID="be840790c395da14e994e5eed7278ff4ab0f9d97d887505937f7ee3cc9f1e3ff" exitCode=0
Oct 29 23:00:37.541092 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:37.540954    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs" event={"ID":"6f1b07f5-c3bd-4382-9318-2234d4e9093e","Type":"ContainerDied","Data":"be840790c395da14e994e5eed7278ff4ab0f9d97d887505937f7ee3cc9f1e3ff"}
Oct 29 23:00:37.541502 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:37.541480    4559 scope.go:122] "RemoveContainer" containerID="be840790c395da14e994e5eed7278ff4ab0f9d97d887505937f7ee3cc9f1e3ff"
Oct 29 23:00:37.543855 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:37.543823    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:00:37.543954 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:37.543897    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:00:37.543954 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:37.543930    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:00:38.581371 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:38.581323    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs" event={"ID":"6f1b07f5-c3bd-4382-9318-2234d4e9093e","Type":"ContainerStarted","Data":"64b6131ecbb0b458ec41c28aa33d1bafb771bef511c245a8fa796e68001e8c5d"}
Oct 29 23:00:38.821920 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:38.821890    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"
Oct 29 23:00:38.983647 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:38.983600    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/199dffea-70ca-4137-90f7-9349eeac1660-kube-api-access-rtcxl\" (UniqueName: \"kubernetes.io/projected/199dffea-70ca-4137-90f7-9349eeac1660-kube-api-access-rtcxl\") pod \"199dffea-70ca-4137-90f7-9349eeac1660\" (UID: \"199dffea-70ca-4137-90f7-9349eeac1660\") "
Oct 29 23:00:38.983879 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:38.983775    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/199dffea-70ca-4137-90f7-9349eeac1660-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/199dffea-70ca-4137-90f7-9349eeac1660-maas-api-openshift-ca\") pod \"199dffea-70ca-4137-90f7-9349eeac1660\" (UID: \"199dffea-70ca-4137-90f7-9349eeac1660\") "
Oct 29 23:00:38.983879 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:38.983843    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/199dffea-70ca-4137-90f7-9349eeac1660-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/199dffea-70ca-4137-90f7-9349eeac1660-maas-api-xks-ca\") pod \"199dffea-70ca-4137-90f7-9349eeac1660\" (UID: \"199dffea-70ca-4137-90f7-9349eeac1660\") "
Oct 29 23:00:38.984893 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:38.984860    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/199dffea-70ca-4137-90f7-9349eeac1660-maas-api-openshift-ca" pod "199dffea-70ca-4137-90f7-9349eeac1660" (UID: "199dffea-70ca-4137-90f7-9349eeac1660"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:38.986444 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:38.986408    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199dffea-70ca-4137-90f7-9349eeac1660-kube-api-access-rtcxl" pod "199dffea-70ca-4137-90f7-9349eeac1660" (UID: "199dffea-70ca-4137-90f7-9349eeac1660"). InnerVolumeSpecName "kube-api-access-rtcxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:38.987019 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:38.986992    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/199dffea-70ca-4137-90f7-9349eeac1660-maas-api-xks-ca" pod "199dffea-70ca-4137-90f7-9349eeac1660" (UID: "199dffea-70ca-4137-90f7-9349eeac1660"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:39.086504 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:39.086462    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/199dffea-70ca-4137-90f7-9349eeac1660-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:39.086504 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:39.086506    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtcxl\" (UniqueName: \"kubernetes.io/projected/199dffea-70ca-4137-90f7-9349eeac1660-kube-api-access-rtcxl\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:39.086717 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:39.086523    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/199dffea-70ca-4137-90f7-9349eeac1660-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:39.483446 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:39.483412    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 23:00:39.483872 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:39.483835    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Oct 29 23:00:39.484126 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:39.484089    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs" podUID="6f1b07f5-c3bd-4382-9318-2234d4e9093e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Oct 29 23:00:39.596854 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:39.596717    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg" event={"ID":"199dffea-70ca-4137-90f7-9349eeac1660","Type":"ContainerDied","Data":"901a8570e36b50953620bc4c9515ac2608f25aa036ff15fead67d93e1dc04718"}
Oct 29 23:00:39.596854 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:39.596778    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="901a8570e36b50953620bc4c9515ac2608f25aa036ff15fead67d93e1dc04718"
Oct 29 23:00:39.597488 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:39.596801    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"
Oct 29 23:00:39.617239 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:39.617196    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d" podUID="851f7422-65c1-4771-b581-cc578a3ce43f" containerName="payload-processing" containerID="cri-o://5cec1ddf5dea9ce6dcaa1d9b5d7eeb1c8303847c5ff00f743e4abc991119c741" gracePeriod=25
Oct 29 23:00:39.656204 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:39.652166    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc" podUID="c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40" containerName="payload-pre-processing" containerID="cri-o://b486bdc2091ec8f5d2b6813409aee037344b2cf17c7daa6b13c7f65094ac1d67" gracePeriod=25
Oct 29 23:00:40.089672 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.089643    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"
Oct 29 23:00:40.127291 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.127220    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"
Oct 29 23:00:40.183291 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.181647    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40-plugins-config-volume\") pod \"c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40\" (UID: \"c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40\") "
Oct 29 23:00:40.183291 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.181837    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/851f7422-65c1-4771-b581-cc578a3ce43f-kube-api-access-m6r9z\" (UniqueName: \"kubernetes.io/projected/851f7422-65c1-4771-b581-cc578a3ce43f-kube-api-access-m6r9z\") pod \"851f7422-65c1-4771-b581-cc578a3ce43f\" (UID: \"851f7422-65c1-4771-b581-cc578a3ce43f\") "
Oct 29 23:00:40.183291 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.181888    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/851f7422-65c1-4771-b581-cc578a3ce43f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/851f7422-65c1-4771-b581-cc578a3ce43f-plugins-config-volume\") pod \"851f7422-65c1-4771-b581-cc578a3ce43f\" (UID: \"851f7422-65c1-4771-b581-cc578a3ce43f\") "
Oct 29 23:00:40.183291 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.181923    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40-kube-api-access-2prd2\" (UniqueName: \"kubernetes.io/projected/c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40-kube-api-access-2prd2\") pod \"c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40\" (UID: \"c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40\") "
Oct 29 23:00:40.183291 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.183222    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851f7422-65c1-4771-b581-cc578a3ce43f-plugins-config-volume" pod "851f7422-65c1-4771-b581-cc578a3ce43f" (UID: "851f7422-65c1-4771-b581-cc578a3ce43f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:40.183943 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.183570    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40-plugins-config-volume" pod "c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40" (UID: "c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:40.185351 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.185321    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851f7422-65c1-4771-b581-cc578a3ce43f-kube-api-access-m6r9z" pod "851f7422-65c1-4771-b581-cc578a3ce43f" (UID: "851f7422-65c1-4771-b581-cc578a3ce43f"). InnerVolumeSpecName "kube-api-access-m6r9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:40.186669 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.186636    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40-kube-api-access-2prd2" pod "c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40" (UID: "c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40"). InnerVolumeSpecName "kube-api-access-2prd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:40.284607 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.284559    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6r9z\" (UniqueName: \"kubernetes.io/projected/851f7422-65c1-4771-b581-cc578a3ce43f-kube-api-access-m6r9z\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:40.284865 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.284847    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/851f7422-65c1-4771-b581-cc578a3ce43f-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:40.284997 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.284985    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2prd2\" (UniqueName: \"kubernetes.io/projected/c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40-kube-api-access-2prd2\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:40.285096 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.285085    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:40.482475 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.482429    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Oct 29 23:00:40.482703 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.482492    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs" podUID="6f1b07f5-c3bd-4382-9318-2234d4e9093e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Oct 29 23:00:40.607423 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.607390    4559 generic.go:356] "Generic (PLEG): container finished" podID="c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40" containerID="b486bdc2091ec8f5d2b6813409aee037344b2cf17c7daa6b13c7f65094ac1d67" exitCode=0
Oct 29 23:00:40.607922 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.607437    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"
Oct 29 23:00:40.607922 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.607555    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc" event={"ID":"c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40","Type":"ContainerDied","Data":"b486bdc2091ec8f5d2b6813409aee037344b2cf17c7daa6b13c7f65094ac1d67"}
Oct 29 23:00:40.607922 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.607596    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc" event={"ID":"c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40","Type":"ContainerDied","Data":"a7f907c6e2c60dc7c9449116a2979ac4f262c6a95b142672a26959ef5c10c27b"}
Oct 29 23:00:40.607922 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.607618    4559 scope.go:122] "RemoveContainer" containerID="b486bdc2091ec8f5d2b6813409aee037344b2cf17c7daa6b13c7f65094ac1d67"
Oct 29 23:00:40.609705 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.609678    4559 generic.go:356] "Generic (PLEG): container finished" podID="851f7422-65c1-4771-b581-cc578a3ce43f" containerID="5cec1ddf5dea9ce6dcaa1d9b5d7eeb1c8303847c5ff00f743e4abc991119c741" exitCode=0
Oct 29 23:00:40.609966 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.609947    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d" event={"ID":"851f7422-65c1-4771-b581-cc578a3ce43f","Type":"ContainerDied","Data":"5cec1ddf5dea9ce6dcaa1d9b5d7eeb1c8303847c5ff00f743e4abc991119c741"}
Oct 29 23:00:40.610036 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.609986    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d" event={"ID":"851f7422-65c1-4771-b581-cc578a3ce43f","Type":"ContainerDied","Data":"90991d519e3faa49d5993daf233b8c1e186feba46610785688de743023283999"}
Oct 29 23:00:40.610075 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.610039    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"
Oct 29 23:00:40.622429 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.622258    4559 scope.go:122] "RemoveContainer" containerID="b486bdc2091ec8f5d2b6813409aee037344b2cf17c7daa6b13c7f65094ac1d67"
Oct 29 23:00:40.622604 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:40.622579    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b486bdc2091ec8f5d2b6813409aee037344b2cf17c7daa6b13c7f65094ac1d67\": container with ID starting with b486bdc2091ec8f5d2b6813409aee037344b2cf17c7daa6b13c7f65094ac1d67 not found: ID does not exist" containerID="b486bdc2091ec8f5d2b6813409aee037344b2cf17c7daa6b13c7f65094ac1d67"
Oct 29 23:00:40.622686 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.622614    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b486bdc2091ec8f5d2b6813409aee037344b2cf17c7daa6b13c7f65094ac1d67"} err="failed to get container status \"b486bdc2091ec8f5d2b6813409aee037344b2cf17c7daa6b13c7f65094ac1d67\": rpc error: code = NotFound desc = could not find container \"b486bdc2091ec8f5d2b6813409aee037344b2cf17c7daa6b13c7f65094ac1d67\": container with ID starting with b486bdc2091ec8f5d2b6813409aee037344b2cf17c7daa6b13c7f65094ac1d67 not found: ID does not exist"
Oct 29 23:00:40.622686 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.622631    4559 scope.go:122] "RemoveContainer" containerID="5cec1ddf5dea9ce6dcaa1d9b5d7eeb1c8303847c5ff00f743e4abc991119c741"
Oct 29 23:00:40.643946 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.642974    4559 scope.go:122] "RemoveContainer" containerID="5cec1ddf5dea9ce6dcaa1d9b5d7eeb1c8303847c5ff00f743e4abc991119c741"
Oct 29 23:00:40.644606 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:40.644522    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cec1ddf5dea9ce6dcaa1d9b5d7eeb1c8303847c5ff00f743e4abc991119c741\": container with ID starting with 5cec1ddf5dea9ce6dcaa1d9b5d7eeb1c8303847c5ff00f743e4abc991119c741 not found: ID does not exist" containerID="5cec1ddf5dea9ce6dcaa1d9b5d7eeb1c8303847c5ff00f743e4abc991119c741"
Oct 29 23:00:40.644606 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.644565    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cec1ddf5dea9ce6dcaa1d9b5d7eeb1c8303847c5ff00f743e4abc991119c741"} err="failed to get container status \"5cec1ddf5dea9ce6dcaa1d9b5d7eeb1c8303847c5ff00f743e4abc991119c741\": rpc error: code = NotFound desc = could not find container \"5cec1ddf5dea9ce6dcaa1d9b5d7eeb1c8303847c5ff00f743e4abc991119c741\": container with ID starting with 5cec1ddf5dea9ce6dcaa1d9b5d7eeb1c8303847c5ff00f743e4abc991119c741 not found: ID does not exist"
Oct 29 23:00:40.671547 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.671497    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"]
Oct 29 23:00:40.678195 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.678147    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-80248b-79b8c55464-d7k5d"]
Oct 29 23:00:40.715656 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.715619    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"]
Oct 29 23:00:40.722774 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:40.722735    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5bd96b4c59-zqqfc"]
Oct 29 23:00:41.090256 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.087040    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"]
Oct 29 23:00:41.090256 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.087300    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh" podUID="aa557804-9bd1-4a83-a858-1d150a4ac0fe" containerName="istio-proxy" containerID="cri-o://c7732222d5d1dc5bbda751d8cae745443ec9f2cdb7341b2b716e997e9df1763f" gracePeriod=30
Oct 29 23:00:41.417838 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.403362    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="851f7422-65c1-4771-b581-cc578a3ce43f" path="/var/lib/kubelet/pods/851f7422-65c1-4771-b581-cc578a3ce43f/volumes"
Oct 29 23:00:41.417838 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.412013    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40" path="/var/lib/kubelet/pods/c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40/volumes"
Oct 29 23:00:41.489833 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.488219    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Oct 29 23:00:41.489833 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.488298    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs" podUID="6f1b07f5-c3bd-4382-9318-2234d4e9093e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Oct 29 23:00:41.627874 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:41.627803    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ae42cd_1e69_44b6_b728_fb34e15e46c7.slice/crio-conmon-81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-conmon-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6a5323_5dab_421e_9c50_b28825aaa895.slice/crio-4b978c63699e8ab4ab4ea9f2ed09240271664b8f578990f0dc9414cf32a8e69c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:00:41.721119 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.721032    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"]
Oct 29 23:00:41.721871 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.721854    4559 cpu_manager.go:414] "containerMap: removing container" podUID="f6fdff4a-3510-4774-8c07-dddad4aa7219" containerName="cleanup"
Oct 29 23:00:41.721928 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.721872    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6fdff4a-3510-4774-8c07-dddad4aa7219" containerName="cleanup"
Oct 29 23:00:41.721928 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.721889    4559 cpu_manager.go:414] "containerMap: removing container" podUID="40c6ad42-57b2-41c2-a6a5-30fa68a808d4" containerName="cleanup"
Oct 29 23:00:41.721928 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.721894    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c6ad42-57b2-41c2-a6a5-30fa68a808d4" containerName="cleanup"
Oct 29 23:00:41.721928 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.721905    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7498470d-5bab-4716-8feb-c0e70788cc88" containerName="cleanup"
Oct 29 23:00:41.721928 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.721910    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7498470d-5bab-4716-8feb-c0e70788cc88" containerName="cleanup"
Oct 29 23:00:41.721928 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.721926    4559 cpu_manager.go:414] "containerMap: removing container" podUID="199dffea-70ca-4137-90f7-9349eeac1660" containerName="revoke-keys"
Oct 29 23:00:41.722121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.721930    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="199dffea-70ca-4137-90f7-9349eeac1660" containerName="revoke-keys"
Oct 29 23:00:41.722121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.721948    4559 cpu_manager.go:414] "containerMap: removing container" podUID="851f7422-65c1-4771-b581-cc578a3ce43f" containerName="payload-processing"
Oct 29 23:00:41.722121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.721952    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="851f7422-65c1-4771-b581-cc578a3ce43f" containerName="payload-processing"
Oct 29 23:00:41.722121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.721960    4559 cpu_manager.go:414] "containerMap: removing container" podUID="1423563c-a109-46a3-9fd5-cccab25939d3" containerName="istio-proxy"
Oct 29 23:00:41.722121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.721964    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1423563c-a109-46a3-9fd5-cccab25939d3" containerName="istio-proxy"
Oct 29 23:00:41.722121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.721985    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bb6a5323-5dab-421e-9c50-b28825aaa895" containerName="cleanup"
Oct 29 23:00:41.722121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.721989    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6a5323-5dab-421e-9c50-b28825aaa895" containerName="cleanup"
Oct 29 23:00:41.722121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722001    4559 cpu_manager.go:414] "containerMap: removing container" podUID="58b25906-470a-4dbd-b43a-0f0bcd25223c" containerName="cleanup"
Oct 29 23:00:41.722121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722007    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58b25906-470a-4dbd-b43a-0f0bcd25223c" containerName="cleanup"
Oct 29 23:00:41.722121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722012    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3b1c42ea-be0a-4b54-b41c-cd7116493b54" containerName="cleanup"
Oct 29 23:00:41.722121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722016    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1c42ea-be0a-4b54-b41c-cd7116493b54" containerName="cleanup"
Oct 29 23:00:41.722121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722028    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40" containerName="payload-pre-processing"
Oct 29 23:00:41.722121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722033    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40" containerName="payload-pre-processing"
Oct 29 23:00:41.722121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722044    4559 cpu_manager.go:414] "containerMap: removing container" podUID="619b371d-35a7-43d9-ace8-cfc478f2fdfa" containerName="cleanup"
Oct 29 23:00:41.722121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722048    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619b371d-35a7-43d9-ace8-cfc478f2fdfa" containerName="cleanup"
Oct 29 23:00:41.722573 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722155    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="851f7422-65c1-4771-b581-cc578a3ce43f" containerName="payload-processing"
Oct 29 23:00:41.722573 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722165    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="1423563c-a109-46a3-9fd5-cccab25939d3" containerName="istio-proxy"
Oct 29 23:00:41.722573 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722174    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1c42ea-be0a-4b54-b41c-cd7116493b54" containerName="cleanup"
Oct 29 23:00:41.722573 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722182    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c6ad42-57b2-41c2-a6a5-30fa68a808d4" containerName="cleanup"
Oct 29 23:00:41.722573 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722188    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7498470d-5bab-4716-8feb-c0e70788cc88" containerName="cleanup"
Oct 29 23:00:41.722573 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722193    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="58b25906-470a-4dbd-b43a-0f0bcd25223c" containerName="cleanup"
Oct 29 23:00:41.722573 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722200    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c460b9f9-ba2f-4acf-9d2f-8ea1dea0ce40" containerName="payload-pre-processing"
Oct 29 23:00:41.722573 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722205    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ea5735-9ca9-4980-8031-44fe6c4269ef" containerName="cleanup"
Oct 29 23:00:41.722573 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722210    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde5a64f-e410-46b8-8e6f-c2933360770a" containerName="istio-proxy"
Oct 29 23:00:41.722573 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722216    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="619b371d-35a7-43d9-ace8-cfc478f2fdfa" containerName="cleanup"
Oct 29 23:00:41.722573 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722222    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6fdff4a-3510-4774-8c07-dddad4aa7219" containerName="cleanup"
Oct 29 23:00:41.722573 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.722231    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="199dffea-70ca-4137-90f7-9349eeac1660" containerName="revoke-keys"
Oct 29 23:00:41.726301 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.726281    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"
Oct 29 23:00:41.775774 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.775738    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:00:41.775996 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.775832    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:00:41.778039 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.778011    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"]
Oct 29 23:00:41.874928 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.874889    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86drv\" (UniqueName: \"kubernetes.io/projected/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-kube-api-access-86drv\") pod \"maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln\" (UID: \"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"
Oct 29 23:00:41.875310 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.875283    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln\" (UID: \"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"
Oct 29 23:00:41.875386 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.875352    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln\" (UID: \"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"
Oct 29 23:00:41.976695 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.976584    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86drv\" (UniqueName: \"kubernetes.io/projected/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-kube-api-access-86drv\") pod \"maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln\" (UID: \"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"
Oct 29 23:00:41.977275 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.976799    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln\" (UID: \"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"
Oct 29 23:00:41.977275 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.976853    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln\" (UID: \"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"
Oct 29 23:00:41.977515 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.977492    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln\" (UID: \"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"
Oct 29 23:00:41.981854 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:41.979955    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln\" (UID: \"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"
Oct 29 23:00:42.000057 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:42.000025    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86drv\" (UniqueName: \"kubernetes.io/projected/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-kube-api-access-86drv\") pod \"maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln\" (UID: \"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"
Oct 29 23:00:42.078213 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:42.078179    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"
Oct 29 23:00:42.281978 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:42.280770    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"]
Oct 29 23:00:42.281978 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:42.281018    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k" podUID="faec6b9a-7468-48ae-8364-1f1913ff7265" containerName="istio-proxy" containerID="cri-o://a481b7088306fa00a66ac1e1c467a4a7cc8392db788b0e7a23a078ec73539471" gracePeriod=30
Oct 29 23:00:42.431033 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:00:42.430998    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f03ba23_9c3b_4e0d_bb48_c5efe44e4cf3.slice/crio-423feb2d3cab59adba9400203f65065b194f384ad9cd34714ac52b60b8ab3070 WatchSource:0}: Error finding container 423feb2d3cab59adba9400203f65065b194f384ad9cd34714ac52b60b8ab3070: Status 404 returned error can't find the container with id 423feb2d3cab59adba9400203f65065b194f384ad9cd34714ac52b60b8ab3070
Oct 29 23:00:42.443849 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:42.433774    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"]
Oct 29 23:00:42.482565 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:42.482342    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Oct 29 23:00:42.482565 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:42.482392    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs" podUID="6f1b07f5-c3bd-4382-9318-2234d4e9093e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Oct 29 23:00:42.632777 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:42.632685    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln" event={"ID":"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3","Type":"ContainerStarted","Data":"423feb2d3cab59adba9400203f65065b194f384ad9cd34714ac52b60b8ab3070"}
Oct 29 23:00:43.485935 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.482694    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Oct 29 23:00:43.485935 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.482750    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs" podUID="6f1b07f5-c3bd-4382-9318-2234d4e9093e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Oct 29 23:00:43.703580 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.703494    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln" event={"ID":"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3","Type":"ContainerStarted","Data":"0c8cfd7afc1458df33a8c67bad5d92a3ed882aa2c0dee5cf8e25ba2dc4b439d7"}
Oct 29 23:00:43.706974 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.706934    4559 generic.go:356] "Generic (PLEG): container finished" podID="2b2f4588-6420-4161-a8f1-f7a888ff1993" containerID="a3f8df9d925f6459ed1d048e28ee3603a78a1727d6509bd2eb98cd46806fd44a" exitCode=0
Oct 29 23:00:43.707161 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.707050    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg" event={"ID":"2b2f4588-6420-4161-a8f1-f7a888ff1993","Type":"ContainerDied","Data":"a3f8df9d925f6459ed1d048e28ee3603a78a1727d6509bd2eb98cd46806fd44a"}
Oct 29 23:00:43.709318 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.709283    4559 generic.go:356] "Generic (PLEG): container finished" podID="11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24" containerID="acbc176d1b91b36a2279837f7589acc4c0d3c18de463405ac1730b4c5deb7a99" exitCode=0
Oct 29 23:00:43.709587 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.709554    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z" event={"ID":"11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24","Type":"ContainerDied","Data":"acbc176d1b91b36a2279837f7589acc4c0d3c18de463405ac1730b4c5deb7a99"}
Oct 29 23:00:43.727238 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:43.727210    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:00:43.728767 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.728742    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"
Oct 29 23:00:43.789465 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.789437    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"
Oct 29 23:00:43.852426 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.848444    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24-plugins-config-volume\") pod \"11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24\" (UID: \"11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24\") "
Oct 29 23:00:43.852426 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.848649    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b2f4588-6420-4161-a8f1-f7a888ff1993-kube-api-access-h5xf2\" (UniqueName: \"kubernetes.io/projected/2b2f4588-6420-4161-a8f1-f7a888ff1993-kube-api-access-h5xf2\") pod \"2b2f4588-6420-4161-a8f1-f7a888ff1993\" (UID: \"2b2f4588-6420-4161-a8f1-f7a888ff1993\") "
Oct 29 23:00:43.852426 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.848696    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24-kube-api-access-85956\" (UniqueName: \"kubernetes.io/projected/11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24-kube-api-access-85956\") pod \"11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24\" (UID: \"11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24\") "
Oct 29 23:00:43.852426 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.848887    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b2f4588-6420-4161-a8f1-f7a888ff1993-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2f4588-6420-4161-a8f1-f7a888ff1993-plugins-config-volume\") pod \"2b2f4588-6420-4161-a8f1-f7a888ff1993\" (UID: \"2b2f4588-6420-4161-a8f1-f7a888ff1993\") "
Oct 29 23:00:43.852426 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.850008    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2f4588-6420-4161-a8f1-f7a888ff1993-plugins-config-volume" pod "2b2f4588-6420-4161-a8f1-f7a888ff1993" (UID: "2b2f4588-6420-4161-a8f1-f7a888ff1993"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:43.852426 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.850637    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24-plugins-config-volume" pod "11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24" (UID: "11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:43.863683 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.859301    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24-kube-api-access-85956" pod "11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24" (UID: "11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24"). InnerVolumeSpecName "kube-api-access-85956". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:43.868623 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.867111    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2f4588-6420-4161-a8f1-f7a888ff1993-kube-api-access-h5xf2" pod "2b2f4588-6420-4161-a8f1-f7a888ff1993" (UID: "2b2f4588-6420-4161-a8f1-f7a888ff1993"). InnerVolumeSpecName "kube-api-access-h5xf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:43.951415 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.951314    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:43.951415 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.951349    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5xf2\" (UniqueName: \"kubernetes.io/projected/2b2f4588-6420-4161-a8f1-f7a888ff1993-kube-api-access-h5xf2\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:43.951415 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.951361    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85956\" (UniqueName: \"kubernetes.io/projected/11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24-kube-api-access-85956\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:43.951415 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:43.951375    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2f4588-6420-4161-a8f1-f7a888ff1993-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.487980 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:44.487947    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 23:00:44.488419 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:44.488393    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 23:00:44.489307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:44.489286    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 23:00:44.779618 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:44.779486    4559 generic.go:356] "Generic (PLEG): container finished" podID="5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3" containerID="0c8cfd7afc1458df33a8c67bad5d92a3ed882aa2c0dee5cf8e25ba2dc4b439d7" exitCode=0
Oct 29 23:00:44.779618 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:44.779563    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln" event={"ID":"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3","Type":"ContainerDied","Data":"0c8cfd7afc1458df33a8c67bad5d92a3ed882aa2c0dee5cf8e25ba2dc4b439d7"}
Oct 29 23:00:44.782608 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:44.782579    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg" event={"ID":"2b2f4588-6420-4161-a8f1-f7a888ff1993","Type":"ContainerDied","Data":"e06b9364c9442dea250d8dfdb6a28511032fc11b140a03727fc2c1a9b9634458"}
Oct 29 23:00:44.782779 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:44.782643    4559 scope.go:122] "RemoveContainer" containerID="a3f8df9d925f6459ed1d048e28ee3603a78a1727d6509bd2eb98cd46806fd44a"
Oct 29 23:00:44.782933 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:44.782595    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"
Oct 29 23:00:44.785074 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:44.784954    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z" event={"ID":"11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24","Type":"ContainerDied","Data":"7f647527d2b7e063876839ccd6f75f45ec64bfe523af9359ee097a72d8daae38"}
Oct 29 23:00:44.785452 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:44.785322    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"
Oct 29 23:00:44.800770 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:44.800742    4559 scope.go:122] "RemoveContainer" containerID="acbc176d1b91b36a2279837f7589acc4c0d3c18de463405ac1730b4c5deb7a99"
Oct 29 23:00:44.844044 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:44.844017    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"]
Oct 29 23:00:44.848057 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:44.848027    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c933cb-786d7ff8b7-zhqxg"]
Oct 29 23:00:44.885429 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:44.885373    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"]
Oct 29 23:00:44.940898 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:44.940853    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c933cb-fc7cf9456-g8t8z"]
Oct 29 23:00:45.001430 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:45.001400    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"
Oct 29 23:00:45.090478 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:45.090377    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-kube-api-access-86drv\" (UniqueName: \"kubernetes.io/projected/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-kube-api-access-86drv\") pod \"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3\" (UID: \"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3\") "
Oct 29 23:00:45.090667 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:45.090583    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-maas-api-openshift-ca\") pod \"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3\" (UID: \"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3\") "
Oct 29 23:00:45.090749 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:45.090685    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-maas-api-xks-ca\") pod \"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3\" (UID: \"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3\") "
Oct 29 23:00:45.091399 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:45.091365    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-maas-api-openshift-ca" pod "5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3" (UID: "5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:45.093496 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:45.093471    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-maas-api-xks-ca" pod "5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3" (UID: "5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:45.093759 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:45.093722    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-kube-api-access-86drv" pod "5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3" (UID: "5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3"). InnerVolumeSpecName "kube-api-access-86drv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:45.193104 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:45.193064    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:45.193104 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:45.193098    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:45.193104 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:45.193109    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86drv\" (UniqueName: \"kubernetes.io/projected/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3-kube-api-access-86drv\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:45.231820 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:45.231770    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24" path="/var/lib/kubelet/pods/11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24/volumes"
Oct 29 23:00:45.232326 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:45.232308    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b2f4588-6420-4161-a8f1-f7a888ff1993" path="/var/lib/kubelet/pods/2b2f4588-6420-4161-a8f1-f7a888ff1993/volumes"
Oct 29 23:00:45.794224 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:45.794188    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln" event={"ID":"5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3","Type":"ContainerDied","Data":"423feb2d3cab59adba9400203f65065b194f384ad9cd34714ac52b60b8ab3070"}
Oct 29 23:00:45.794661 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:45.794249    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="423feb2d3cab59adba9400203f65065b194f384ad9cd34714ac52b60b8ab3070"
Oct 29 23:00:45.794661 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:45.794197    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"
Oct 29 23:00:46.409741 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.409708    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 23:00:46.517247 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.517208    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa557804-9bd1-4a83-a858-1d150a4ac0fe-kube-api-access-8fq72\" (UniqueName: \"kubernetes.io/projected/aa557804-9bd1-4a83-a858-1d150a4ac0fe-kube-api-access-8fq72\") pod \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") "
Oct 29 23:00:46.517590 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.517553    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-workload-socket\") pod \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") "
Oct 29 23:00:46.517716 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.517626    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-data\") pod \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") "
Oct 29 23:00:46.517716 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.517698    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-credential-socket\") pod \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") "
Oct 29 23:00:46.517888 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.517794    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-envoy\") pod \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") "
Oct 29 23:00:46.517888 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.517795    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-workload-socket" pod "aa557804-9bd1-4a83-a858-1d150a4ac0fe" (UID: "aa557804-9bd1-4a83-a858-1d150a4ac0fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.517888 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.517872    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istiod-ca-cert\") pod \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") "
Oct 29 23:00:46.518073 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.517965    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-workload-certs\") pod \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") "
Oct 29 23:00:46.518073 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.517992    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-credential-socket" pod "aa557804-9bd1-4a83-a858-1d150a4ac0fe" (UID: "aa557804-9bd1-4a83-a858-1d150a4ac0fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.518188 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.518093    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-podinfo\") pod \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") "
Oct 29 23:00:46.518188 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.518176    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-token\" (UniqueName: \"kubernetes.io/projected/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-token\") pod \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\" (UID: \"aa557804-9bd1-4a83-a858-1d150a4ac0fe\") "
Oct 29 23:00:46.518305 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.518279    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-workload-certs" pod "aa557804-9bd1-4a83-a858-1d150a4ac0fe" (UID: "aa557804-9bd1-4a83-a858-1d150a4ac0fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.518372 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.518314    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istiod-ca-cert" pod "aa557804-9bd1-4a83-a858-1d150a4ac0fe" (UID: "aa557804-9bd1-4a83-a858-1d150a4ac0fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:46.518651 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.518626    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-data" pod "aa557804-9bd1-4a83-a858-1d150a4ac0fe" (UID: "aa557804-9bd1-4a83-a858-1d150a4ac0fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.518957 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.518937    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.519071 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.518961    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.519071 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.518973    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.519071 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.518985    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.519071 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.518999    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.521777 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.521748    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-envoy" pod "aa557804-9bd1-4a83-a858-1d150a4ac0fe" (UID: "aa557804-9bd1-4a83-a858-1d150a4ac0fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.521777 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.521754    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-token" pod "aa557804-9bd1-4a83-a858-1d150a4ac0fe" (UID: "aa557804-9bd1-4a83-a858-1d150a4ac0fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:46.522001 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.521924    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa557804-9bd1-4a83-a858-1d150a4ac0fe-kube-api-access-8fq72" pod "aa557804-9bd1-4a83-a858-1d150a4ac0fe" (UID: "aa557804-9bd1-4a83-a858-1d150a4ac0fe"). InnerVolumeSpecName "kube-api-access-8fq72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:46.522635 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.522608    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-podinfo" pod "aa557804-9bd1-4a83-a858-1d150a4ac0fe" (UID: "aa557804-9bd1-4a83-a858-1d150a4ac0fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:46.564137 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.564092    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"]
Oct 29 23:00:46.620320 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.620275    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.620320 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.620311    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.620320 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.620322    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fq72\" (UniqueName: \"kubernetes.io/projected/aa557804-9bd1-4a83-a858-1d150a4ac0fe-kube-api-access-8fq72\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.620320 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.620331    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa557804-9bd1-4a83-a858-1d150a4ac0fe-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.867606 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.867573    4559 generic.go:356] "Generic (PLEG): container finished" podID="aa557804-9bd1-4a83-a858-1d150a4ac0fe" containerID="c7732222d5d1dc5bbda751d8cae745443ec9f2cdb7341b2b716e997e9df1763f" exitCode=0
Oct 29 23:00:46.868160 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.867639    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"
Oct 29 23:00:46.868160 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.867661    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh" event={"ID":"aa557804-9bd1-4a83-a858-1d150a4ac0fe","Type":"ContainerDied","Data":"c7732222d5d1dc5bbda751d8cae745443ec9f2cdb7341b2b716e997e9df1763f"}
Oct 29 23:00:46.868160 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.867719    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh" event={"ID":"aa557804-9bd1-4a83-a858-1d150a4ac0fe","Type":"ContainerDied","Data":"14618c2c4d2e89e8ee60da9acc09a4ec6971c687d308b57985bf4dd6d9861480"}
Oct 29 23:00:46.868160 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.867743    4559 scope.go:122] "RemoveContainer" containerID="c7732222d5d1dc5bbda751d8cae745443ec9f2cdb7341b2b716e997e9df1763f"
Oct 29 23:00:46.882538 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.882517    4559 scope.go:122] "RemoveContainer" containerID="433dae1e18022d12eed9c784849ede4540a8087123ec41a216b1089dc8d76f7d"
Oct 29 23:00:46.896556 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.896524    4559 scope.go:122] "RemoveContainer" containerID="c7732222d5d1dc5bbda751d8cae745443ec9f2cdb7341b2b716e997e9df1763f"
Oct 29 23:00:46.896970 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:46.896950    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7732222d5d1dc5bbda751d8cae745443ec9f2cdb7341b2b716e997e9df1763f\": container with ID starting with c7732222d5d1dc5bbda751d8cae745443ec9f2cdb7341b2b716e997e9df1763f not found: ID does not exist" containerID="c7732222d5d1dc5bbda751d8cae745443ec9f2cdb7341b2b716e997e9df1763f"
Oct 29 23:00:46.897075 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.896981    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7732222d5d1dc5bbda751d8cae745443ec9f2cdb7341b2b716e997e9df1763f"} err="failed to get container status \"c7732222d5d1dc5bbda751d8cae745443ec9f2cdb7341b2b716e997e9df1763f\": rpc error: code = NotFound desc = could not find container \"c7732222d5d1dc5bbda751d8cae745443ec9f2cdb7341b2b716e997e9df1763f\": container with ID starting with c7732222d5d1dc5bbda751d8cae745443ec9f2cdb7341b2b716e997e9df1763f not found: ID does not exist"
Oct 29 23:00:46.897075 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.896999    4559 scope.go:122] "RemoveContainer" containerID="433dae1e18022d12eed9c784849ede4540a8087123ec41a216b1089dc8d76f7d"
Oct 29 23:00:46.897328 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:46.897299    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"433dae1e18022d12eed9c784849ede4540a8087123ec41a216b1089dc8d76f7d\": container with ID starting with 433dae1e18022d12eed9c784849ede4540a8087123ec41a216b1089dc8d76f7d not found: ID does not exist" containerID="433dae1e18022d12eed9c784849ede4540a8087123ec41a216b1089dc8d76f7d"
Oct 29 23:00:46.897416 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.897333    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433dae1e18022d12eed9c784849ede4540a8087123ec41a216b1089dc8d76f7d"} err="failed to get container status \"433dae1e18022d12eed9c784849ede4540a8087123ec41a216b1089dc8d76f7d\": rpc error: code = NotFound desc = could not find container \"433dae1e18022d12eed9c784849ede4540a8087123ec41a216b1089dc8d76f7d\": container with ID starting with 433dae1e18022d12eed9c784849ede4540a8087123ec41a216b1089dc8d76f7d not found: ID does not exist"
Oct 29 23:00:46.924798 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.924766    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"]
Oct 29 23:00:46.928844 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:46.928799    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-6mjhh"]
Oct 29 23:00:47.230701 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.230665    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa557804-9bd1-4a83-a858-1d150a4ac0fe" path="/var/lib/kubelet/pods/aa557804-9bd1-4a83-a858-1d150a4ac0fe/volumes"
Oct 29 23:00:47.595015 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.594986    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 23:00:47.756136 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.756094    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-data\") pod \"faec6b9a-7468-48ae-8364-1f1913ff7265\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") "
Oct 29 23:00:47.756389 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.756159    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-credential-socket\") pod \"faec6b9a-7468-48ae-8364-1f1913ff7265\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") "
Oct 29 23:00:47.756389 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.756207    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-workload-certs\") pod \"faec6b9a-7468-48ae-8364-1f1913ff7265\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") "
Oct 29 23:00:47.756389 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.756272    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-podinfo\") pod \"faec6b9a-7468-48ae-8364-1f1913ff7265\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") "
Oct 29 23:00:47.756598 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.756416    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faec6b9a-7468-48ae-8364-1f1913ff7265-kube-api-access-54nb9\" (UniqueName: \"kubernetes.io/projected/faec6b9a-7468-48ae-8364-1f1913ff7265-kube-api-access-54nb9\") pod \"faec6b9a-7468-48ae-8364-1f1913ff7265\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") "
Oct 29 23:00:47.756598 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.756479    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-workload-socket\") pod \"faec6b9a-7468-48ae-8364-1f1913ff7265\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") "
Oct 29 23:00:47.756598 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.756505    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-envoy\") pod \"faec6b9a-7468-48ae-8364-1f1913ff7265\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") "
Oct 29 23:00:47.756763 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.756595    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-token\" (UniqueName: \"kubernetes.io/projected/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-token\") pod \"faec6b9a-7468-48ae-8364-1f1913ff7265\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") "
Oct 29 23:00:47.756763 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.756633    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faec6b9a-7468-48ae-8364-1f1913ff7265-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faec6b9a-7468-48ae-8364-1f1913ff7265-istiod-ca-cert\") pod \"faec6b9a-7468-48ae-8364-1f1913ff7265\" (UID: \"faec6b9a-7468-48ae-8364-1f1913ff7265\") "
Oct 29 23:00:47.756897 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.756828    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-workload-certs" pod "faec6b9a-7468-48ae-8364-1f1913ff7265" (UID: "faec6b9a-7468-48ae-8364-1f1913ff7265"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:47.756957 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.756917    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-workload-socket" pod "faec6b9a-7468-48ae-8364-1f1913ff7265" (UID: "faec6b9a-7468-48ae-8364-1f1913ff7265"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:47.758021 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.757293    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faec6b9a-7468-48ae-8364-1f1913ff7265-istiod-ca-cert" pod "faec6b9a-7468-48ae-8364-1f1913ff7265" (UID: "faec6b9a-7468-48ae-8364-1f1913ff7265"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:47.758021 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.757300    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-credential-socket" pod "faec6b9a-7468-48ae-8364-1f1913ff7265" (UID: "faec6b9a-7468-48ae-8364-1f1913ff7265"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:47.758021 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.757621    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.758021 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.757641    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faec6b9a-7468-48ae-8364-1f1913ff7265-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.758021 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.757655    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.758021 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.757670    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.758021 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.758007    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-data" pod "faec6b9a-7468-48ae-8364-1f1913ff7265" (UID: "faec6b9a-7468-48ae-8364-1f1913ff7265"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:47.759952 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.759923    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-token" pod "faec6b9a-7468-48ae-8364-1f1913ff7265" (UID: "faec6b9a-7468-48ae-8364-1f1913ff7265"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:47.760328 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.760305    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-envoy" pod "faec6b9a-7468-48ae-8364-1f1913ff7265" (UID: "faec6b9a-7468-48ae-8364-1f1913ff7265"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:47.760422 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.760306    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faec6b9a-7468-48ae-8364-1f1913ff7265-kube-api-access-54nb9" pod "faec6b9a-7468-48ae-8364-1f1913ff7265" (UID: "faec6b9a-7468-48ae-8364-1f1913ff7265"). InnerVolumeSpecName "kube-api-access-54nb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:47.760604 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.760582    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-podinfo" pod "faec6b9a-7468-48ae-8364-1f1913ff7265" (UID: "faec6b9a-7468-48ae-8364-1f1913ff7265"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:47.859602 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.859497    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54nb9\" (UniqueName: \"kubernetes.io/projected/faec6b9a-7468-48ae-8364-1f1913ff7265-kube-api-access-54nb9\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.859602 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.859545    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.859602 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.859558    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.859602 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.859566    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.859602 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.859576    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faec6b9a-7468-48ae-8364-1f1913ff7265-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.880475 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.880437    4559 generic.go:356] "Generic (PLEG): container finished" podID="faec6b9a-7468-48ae-8364-1f1913ff7265" containerID="a481b7088306fa00a66ac1e1c467a4a7cc8392db788b0e7a23a078ec73539471" exitCode=0
Oct 29 23:00:47.880970 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.880492    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"
Oct 29 23:00:47.880970 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.880518    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k" event={"ID":"faec6b9a-7468-48ae-8364-1f1913ff7265","Type":"ContainerDied","Data":"a481b7088306fa00a66ac1e1c467a4a7cc8392db788b0e7a23a078ec73539471"}
Oct 29 23:00:47.880970 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.880581    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k" event={"ID":"faec6b9a-7468-48ae-8364-1f1913ff7265","Type":"ContainerDied","Data":"ff1e2cee73998aabe6c6fd8ccf3c27cac298874165ffca884c306e46ca06526c"}
Oct 29 23:00:47.880970 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.880613    4559 scope.go:122] "RemoveContainer" containerID="a481b7088306fa00a66ac1e1c467a4a7cc8392db788b0e7a23a078ec73539471"
Oct 29 23:00:47.893618 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.893592    4559 scope.go:122] "RemoveContainer" containerID="a481b7088306fa00a66ac1e1c467a4a7cc8392db788b0e7a23a078ec73539471"
Oct 29 23:00:47.893964 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:47.893935    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a481b7088306fa00a66ac1e1c467a4a7cc8392db788b0e7a23a078ec73539471\": container with ID starting with a481b7088306fa00a66ac1e1c467a4a7cc8392db788b0e7a23a078ec73539471 not found: ID does not exist" containerID="a481b7088306fa00a66ac1e1c467a4a7cc8392db788b0e7a23a078ec73539471"
Oct 29 23:00:47.894085 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.893969    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a481b7088306fa00a66ac1e1c467a4a7cc8392db788b0e7a23a078ec73539471"} err="failed to get container status \"a481b7088306fa00a66ac1e1c467a4a7cc8392db788b0e7a23a078ec73539471\": rpc error: code = NotFound desc = could not find container \"a481b7088306fa00a66ac1e1c467a4a7cc8392db788b0e7a23a078ec73539471\": container with ID starting with a481b7088306fa00a66ac1e1c467a4a7cc8392db788b0e7a23a078ec73539471 not found: ID does not exist"
Oct 29 23:00:47.936466 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.936430    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"]
Oct 29 23:00:47.949537 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:47.949500    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-fzz2k"]
Oct 29 23:00:48.557355 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.557318    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"]
Oct 29 23:00:48.558136 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.558118    4559 cpu_manager.go:414] "containerMap: removing container" podUID="faec6b9a-7468-48ae-8364-1f1913ff7265" containerName="istio-proxy"
Oct 29 23:00:48.558219 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.558136    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faec6b9a-7468-48ae-8364-1f1913ff7265" containerName="istio-proxy"
Oct 29 23:00:48.558219 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.558152    4559 cpu_manager.go:414] "containerMap: removing container" podUID="aa557804-9bd1-4a83-a858-1d150a4ac0fe" containerName="istio-proxy"
Oct 29 23:00:48.558219 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.558158    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa557804-9bd1-4a83-a858-1d150a4ac0fe" containerName="istio-proxy"
Oct 29 23:00:48.558219 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.558171    4559 cpu_manager.go:414] "containerMap: removing container" podUID="5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3" containerName="revoke-keys"
Oct 29 23:00:48.558219 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.558177    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3" containerName="revoke-keys"
Oct 29 23:00:48.558219 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.558211    4559 cpu_manager.go:414] "containerMap: removing container" podUID="aa557804-9bd1-4a83-a858-1d150a4ac0fe" containerName="istio-proxy"
Oct 29 23:00:48.558219 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.558215    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa557804-9bd1-4a83-a858-1d150a4ac0fe" containerName="istio-proxy"
Oct 29 23:00:48.558219 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.558222    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2b2f4588-6420-4161-a8f1-f7a888ff1993" containerName="payload-pre-processing"
Oct 29 23:00:48.558631 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.558227    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2f4588-6420-4161-a8f1-f7a888ff1993" containerName="payload-pre-processing"
Oct 29 23:00:48.558631 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.558239    4559 cpu_manager.go:414] "containerMap: removing container" podUID="11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24" containerName="payload-processing"
Oct 29 23:00:48.558631 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.558243    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24" containerName="payload-processing"
Oct 29 23:00:48.558631 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.558337    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c8ba74-8c08-4c3e-a6ac-59cdf7b88d24" containerName="payload-processing"
Oct 29 23:00:48.558631 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.558347    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="faec6b9a-7468-48ae-8364-1f1913ff7265" containerName="istio-proxy"
Oct 29 23:00:48.558631 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.558354    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa557804-9bd1-4a83-a858-1d150a4ac0fe" containerName="istio-proxy"
Oct 29 23:00:48.558631 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.558361    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3" containerName="revoke-keys"
Oct 29 23:00:48.558631 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.558371    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2f4588-6420-4161-a8f1-f7a888ff1993" containerName="payload-pre-processing"
Oct 29 23:00:48.563907 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.563877    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"
Oct 29 23:00:48.582217 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.582179    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"]
Oct 29 23:00:48.588156 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.588125    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:00:48.588400 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.588383    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:00:48.671579 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.671523    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjjrg\" (UniqueName: \"kubernetes.io/projected/a7d971a7-9094-47d7-951b-97c08948c7bd-kube-api-access-wjjrg\") pod \"maas-api-revoke-sub-fd4a109c-87srh\" (UID: \"a7d971a7-9094-47d7-951b-97c08948c7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"
Oct 29 23:00:48.671868 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.671658    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d971a7-9094-47d7-951b-97c08948c7bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd4a109c-87srh\" (UID: \"a7d971a7-9094-47d7-951b-97c08948c7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"
Oct 29 23:00:48.671868 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.671744    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d971a7-9094-47d7-951b-97c08948c7bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd4a109c-87srh\" (UID: \"a7d971a7-9094-47d7-951b-97c08948c7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"
Oct 29 23:00:48.774216 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.774167    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjjrg\" (UniqueName: \"kubernetes.io/projected/a7d971a7-9094-47d7-951b-97c08948c7bd-kube-api-access-wjjrg\") pod \"maas-api-revoke-sub-fd4a109c-87srh\" (UID: \"a7d971a7-9094-47d7-951b-97c08948c7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"
Oct 29 23:00:48.774437 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.774243    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d971a7-9094-47d7-951b-97c08948c7bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd4a109c-87srh\" (UID: \"a7d971a7-9094-47d7-951b-97c08948c7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"
Oct 29 23:00:48.774437 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.774297    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d971a7-9094-47d7-951b-97c08948c7bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd4a109c-87srh\" (UID: \"a7d971a7-9094-47d7-951b-97c08948c7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"
Oct 29 23:00:48.775112 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.775046    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d971a7-9094-47d7-951b-97c08948c7bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd4a109c-87srh\" (UID: \"a7d971a7-9094-47d7-951b-97c08948c7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"
Oct 29 23:00:48.776921 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.776893    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d971a7-9094-47d7-951b-97c08948c7bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd4a109c-87srh\" (UID: \"a7d971a7-9094-47d7-951b-97c08948c7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"
Oct 29 23:00:48.794684 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.794656    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjjrg\" (UniqueName: \"kubernetes.io/projected/a7d971a7-9094-47d7-951b-97c08948c7bd-kube-api-access-wjjrg\") pod \"maas-api-revoke-sub-fd4a109c-87srh\" (UID: \"a7d971a7-9094-47d7-951b-97c08948c7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"
Oct 29 23:00:48.883898 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:48.882064    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"
Oct 29 23:00:49.229522 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:49.229485    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faec6b9a-7468-48ae-8364-1f1913ff7265" path="/var/lib/kubelet/pods/faec6b9a-7468-48ae-8364-1f1913ff7265/volumes"
Oct 29 23:00:49.258430 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:49.258383    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"]
Oct 29 23:00:49.258933 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:00:49.258881    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d971a7_9094_47d7_951b_97c08948c7bd.slice/crio-79d9c5861d15960870624e00e8f6a2e4ea2b6ea329bac60a1b5700a64100c5b4 WatchSource:0}: Error finding container 79d9c5861d15960870624e00e8f6a2e4ea2b6ea329bac60a1b5700a64100c5b4: Status 404 returned error can't find the container with id 79d9c5861d15960870624e00e8f6a2e4ea2b6ea329bac60a1b5700a64100c5b4
Oct 29 23:00:49.957617 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:49.957578    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh" event={"ID":"a7d971a7-9094-47d7-951b-97c08948c7bd","Type":"ContainerStarted","Data":"ca80472e9f25841cbcdc4c09db4ec0169a31d373e2b3bb6435f17d73105ebf85"}
Oct 29 23:00:49.958160 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:49.957661    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh" event={"ID":"a7d971a7-9094-47d7-951b-97c08948c7bd","Type":"ContainerStarted","Data":"79d9c5861d15960870624e00e8f6a2e4ea2b6ea329bac60a1b5700a64100c5b4"}
Oct 29 23:00:50.007781 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:50.007720    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh" podStartSLOduration=2.007705552 podStartE2EDuration="2.007705552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:50.002054443 +0000 UTC m=+2788.913497997" watchObservedRunningTime="2026-10-29 23:00:50.007705552 +0000 UTC m=+2788.919149107"
Oct 29 23:00:50.984772 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:50.984733    4559 generic.go:356] "Generic (PLEG): container finished" podID="a7d971a7-9094-47d7-951b-97c08948c7bd" containerID="ca80472e9f25841cbcdc4c09db4ec0169a31d373e2b3bb6435f17d73105ebf85" exitCode=0
Oct 29 23:00:50.985853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:50.984799    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh" event={"ID":"a7d971a7-9094-47d7-951b-97c08948c7bd","Type":"ContainerDied","Data":"ca80472e9f25841cbcdc4c09db4ec0169a31d373e2b3bb6435f17d73105ebf85"}
Oct 29 23:00:51.232755 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:51.232722    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"]
Oct 29 23:00:51.239846 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:51.239748    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-784e7a-8jhk4"]
Oct 29 23:00:51.946875 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:51.942493    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6a5323_5dab_421e_9c50_b28825aaa895.slice/crio-4b978c63699e8ab4ab4ea9f2ed09240271664b8f578990f0dc9414cf32a8e69c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-conmon-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ae42cd_1e69_44b6_b728_fb34e15e46c7.slice/crio-conmon-81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:00:52.099495 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:52.098944    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6a5323_5dab_421e_9c50_b28825aaa895.slice/crio-4b978c63699e8ab4ab4ea9f2ed09240271664b8f578990f0dc9414cf32a8e69c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ae42cd_1e69_44b6_b728_fb34e15e46c7.slice/crio-conmon-81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-conmon-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:00:52.203524 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:52.203427    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"
Oct 29 23:00:52.371006 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:52.370967    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d971a7-9094-47d7-951b-97c08948c7bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d971a7-9094-47d7-951b-97c08948c7bd-maas-api-openshift-ca\") pod \"a7d971a7-9094-47d7-951b-97c08948c7bd\" (UID: \"a7d971a7-9094-47d7-951b-97c08948c7bd\") "
Oct 29 23:00:52.371382 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:52.371361    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7d971a7-9094-47d7-951b-97c08948c7bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d971a7-9094-47d7-951b-97c08948c7bd-maas-api-xks-ca\") pod \"a7d971a7-9094-47d7-951b-97c08948c7bd\" (UID: \"a7d971a7-9094-47d7-951b-97c08948c7bd\") "
Oct 29 23:00:52.371558 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:52.371541    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d971a7-9094-47d7-951b-97c08948c7bd-kube-api-access-wjjrg\" (UniqueName: \"kubernetes.io/projected/a7d971a7-9094-47d7-951b-97c08948c7bd-kube-api-access-wjjrg\") pod \"a7d971a7-9094-47d7-951b-97c08948c7bd\" (UID: \"a7d971a7-9094-47d7-951b-97c08948c7bd\") "
Oct 29 23:00:52.375586 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:52.375549    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d971a7-9094-47d7-951b-97c08948c7bd-maas-api-openshift-ca" pod "a7d971a7-9094-47d7-951b-97c08948c7bd" (UID: "a7d971a7-9094-47d7-951b-97c08948c7bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:52.376362 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:52.376317    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d971a7-9094-47d7-951b-97c08948c7bd-maas-api-xks-ca" pod "a7d971a7-9094-47d7-951b-97c08948c7bd" (UID: "a7d971a7-9094-47d7-951b-97c08948c7bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:52.413487 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:52.413427    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d971a7-9094-47d7-951b-97c08948c7bd-kube-api-access-wjjrg" pod "a7d971a7-9094-47d7-951b-97c08948c7bd" (UID: "a7d971a7-9094-47d7-951b-97c08948c7bd"). InnerVolumeSpecName "kube-api-access-wjjrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:52.474473 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:52.474382    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d971a7-9094-47d7-951b-97c08948c7bd-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:52.474473 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:52.474414    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d971a7-9094-47d7-951b-97c08948c7bd-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:52.474473 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:52.474424    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjjrg\" (UniqueName: \"kubernetes.io/projected/a7d971a7-9094-47d7-951b-97c08948c7bd-kube-api-access-wjjrg\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:52.994837 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:52.993251    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"]
Oct 29 23:00:53.060941 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.048238    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"]
Oct 29 23:00:53.086837 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.085594    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh" event={"ID":"a7d971a7-9094-47d7-951b-97c08948c7bd","Type":"ContainerDied","Data":"79d9c5861d15960870624e00e8f6a2e4ea2b6ea329bac60a1b5700a64100c5b4"}
Oct 29 23:00:53.086837 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.085706    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79d9c5861d15960870624e00e8f6a2e4ea2b6ea329bac60a1b5700a64100c5b4"
Oct 29 23:00:53.086837 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.085800    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"
Oct 29 23:00:53.229895 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.229855    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74df5d6d-2367-462e-a46e-57feb80833fe" path="/var/lib/kubelet/pods/74df5d6d-2367-462e-a46e-57feb80833fe/volumes"
Oct 29 23:00:53.537357 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.537323    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"]
Oct 29 23:00:53.538300 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.538272    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a7d971a7-9094-47d7-951b-97c08948c7bd" containerName="revoke-keys"
Oct 29 23:00:53.538300 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.538296    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d971a7-9094-47d7-951b-97c08948c7bd" containerName="revoke-keys"
Oct 29 23:00:53.538470 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.538405    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa557804-9bd1-4a83-a858-1d150a4ac0fe" containerName="istio-proxy"
Oct 29 23:00:53.538470 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.538418    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d971a7-9094-47d7-951b-97c08948c7bd" containerName="revoke-keys"
Oct 29 23:00:53.542680 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.542653    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"
Oct 29 23:00:53.556693 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.556653    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"]
Oct 29 23:00:53.569263 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.569233    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:00:53.569453 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.569366    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:00:53.644856 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.644617    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdzwb\" (UniqueName: \"kubernetes.io/projected/11ae1a35-1d04-478b-be37-75c897b761c0-kube-api-access-kdzwb\") pod \"maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp\" (UID: \"11ae1a35-1d04-478b-be37-75c897b761c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"
Oct 29 23:00:53.645164 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.645132    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ae1a35-1d04-478b-be37-75c897b761c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp\" (UID: \"11ae1a35-1d04-478b-be37-75c897b761c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"
Oct 29 23:00:53.645347 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.645326    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ae1a35-1d04-478b-be37-75c897b761c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp\" (UID: \"11ae1a35-1d04-478b-be37-75c897b761c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"
Oct 29 23:00:53.748974 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.748926    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdzwb\" (UniqueName: \"kubernetes.io/projected/11ae1a35-1d04-478b-be37-75c897b761c0-kube-api-access-kdzwb\") pod \"maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp\" (UID: \"11ae1a35-1d04-478b-be37-75c897b761c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"
Oct 29 23:00:53.750075 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.750050    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ae1a35-1d04-478b-be37-75c897b761c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp\" (UID: \"11ae1a35-1d04-478b-be37-75c897b761c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"
Oct 29 23:00:53.751120 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.751099    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ae1a35-1d04-478b-be37-75c897b761c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp\" (UID: \"11ae1a35-1d04-478b-be37-75c897b761c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"
Oct 29 23:00:53.752028 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.751025    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ae1a35-1d04-478b-be37-75c897b761c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp\" (UID: \"11ae1a35-1d04-478b-be37-75c897b761c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"
Oct 29 23:00:53.765433 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.765396    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ae1a35-1d04-478b-be37-75c897b761c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp\" (UID: \"11ae1a35-1d04-478b-be37-75c897b761c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"
Oct 29 23:00:53.788749 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.788642    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdzwb\" (UniqueName: \"kubernetes.io/projected/11ae1a35-1d04-478b-be37-75c897b761c0-kube-api-access-kdzwb\") pod \"maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp\" (UID: \"11ae1a35-1d04-478b-be37-75c897b761c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"
Oct 29 23:00:53.869455 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:53.869406    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"
Oct 29 23:00:54.199203 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:54.199173    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"]
Oct 29 23:00:54.200118 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:00:54.200092    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ae1a35_1d04_478b_be37_75c897b761c0.slice/crio-bffa14c0bb97ee2ef4d3fa0b08ba73cf01ce0d9ca6e1e24a917d9611528dd3bc WatchSource:0}: Error finding container bffa14c0bb97ee2ef4d3fa0b08ba73cf01ce0d9ca6e1e24a917d9611528dd3bc: Status 404 returned error can't find the container with id bffa14c0bb97ee2ef4d3fa0b08ba73cf01ce0d9ca6e1e24a917d9611528dd3bc
Oct 29 23:00:55.117929 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:55.117840    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp" event={"ID":"11ae1a35-1d04-478b-be37-75c897b761c0","Type":"ContainerStarted","Data":"3154d7d58374f93c9ccefd931ae0288d408e75f2dd91da96e71c3315a4dd4998"}
Oct 29 23:00:55.117929 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:55.117896    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp" event={"ID":"11ae1a35-1d04-478b-be37-75c897b761c0","Type":"ContainerStarted","Data":"bffa14c0bb97ee2ef4d3fa0b08ba73cf01ce0d9ca6e1e24a917d9611528dd3bc"}
Oct 29 23:00:55.206798 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:55.206723    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp" podStartSLOduration=2.206696889 podStartE2EDuration="2.206696889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:55.189240922 +0000 UTC m=+2794.100684475" watchObservedRunningTime="2026-10-29 23:00:55.206696889 +0000 UTC m=+2794.118140444"
Oct 29 23:00:56.137847 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:56.130945    4559 generic.go:356] "Generic (PLEG): container finished" podID="11ae1a35-1d04-478b-be37-75c897b761c0" containerID="3154d7d58374f93c9ccefd931ae0288d408e75f2dd91da96e71c3315a4dd4998" exitCode=0
Oct 29 23:00:56.137847 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:56.131032    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp" event={"ID":"11ae1a35-1d04-478b-be37-75c897b761c0","Type":"ContainerDied","Data":"3154d7d58374f93c9ccefd931ae0288d408e75f2dd91da96e71c3315a4dd4998"}
Oct 29 23:00:57.412439 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:57.412408    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"
Oct 29 23:00:57.483030 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:57.482648    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11ae1a35-1d04-478b-be37-75c897b761c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ae1a35-1d04-478b-be37-75c897b761c0-maas-api-xks-ca\") pod \"11ae1a35-1d04-478b-be37-75c897b761c0\" (UID: \"11ae1a35-1d04-478b-be37-75c897b761c0\") "
Oct 29 23:00:57.483030 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:57.482886    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ae1a35-1d04-478b-be37-75c897b761c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ae1a35-1d04-478b-be37-75c897b761c0-maas-api-openshift-ca\") pod \"11ae1a35-1d04-478b-be37-75c897b761c0\" (UID: \"11ae1a35-1d04-478b-be37-75c897b761c0\") "
Oct 29 23:00:57.484667 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:57.483312    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ae1a35-1d04-478b-be37-75c897b761c0-kube-api-access-kdzwb\" (UniqueName: \"kubernetes.io/projected/11ae1a35-1d04-478b-be37-75c897b761c0-kube-api-access-kdzwb\") pod \"11ae1a35-1d04-478b-be37-75c897b761c0\" (UID: \"11ae1a35-1d04-478b-be37-75c897b761c0\") "
Oct 29 23:00:57.484667 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:57.484128    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ae1a35-1d04-478b-be37-75c897b761c0-maas-api-openshift-ca" pod "11ae1a35-1d04-478b-be37-75c897b761c0" (UID: "11ae1a35-1d04-478b-be37-75c897b761c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:57.485786 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:57.485744    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ae1a35-1d04-478b-be37-75c897b761c0-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:57.491926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:57.491886    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ae1a35-1d04-478b-be37-75c897b761c0-kube-api-access-kdzwb" pod "11ae1a35-1d04-478b-be37-75c897b761c0" (UID: "11ae1a35-1d04-478b-be37-75c897b761c0"). InnerVolumeSpecName "kube-api-access-kdzwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:57.495967 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:57.495938    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ae1a35-1d04-478b-be37-75c897b761c0-maas-api-xks-ca" pod "11ae1a35-1d04-478b-be37-75c897b761c0" (UID: "11ae1a35-1d04-478b-be37-75c897b761c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:57.587284 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:57.587247    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdzwb\" (UniqueName: \"kubernetes.io/projected/11ae1a35-1d04-478b-be37-75c897b761c0-kube-api-access-kdzwb\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:57.587284 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:57.587281    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ae1a35-1d04-478b-be37-75c897b761c0-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:57.994153 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:57.994100    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt" podUID="c2822e0d-87da-4882-a4ed-acebabc7cc8b" containerName="payload-processing" containerID="cri-o://e3ceae8c9809c96c1c4edf12965e9a6f82bd420aa76ca83305d6e6665658bd4c" gracePeriod=25
Oct 29 23:00:58.064003 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:58.063933    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh" podUID="370fbc49-ac99-41e3-9375-098433ebd4da" containerName="payload-pre-processing" containerID="cri-o://74c9029925c67c596c49882746be99c717f68fa59f9b666e66dbd98fa7f79f99" gracePeriod=25
Oct 29 23:00:58.213966 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:58.213931    4559 generic.go:356] "Generic (PLEG): container finished" podID="c2822e0d-87da-4882-a4ed-acebabc7cc8b" containerID="e3ceae8c9809c96c1c4edf12965e9a6f82bd420aa76ca83305d6e6665658bd4c" exitCode=0
Oct 29 23:00:58.214235 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:58.214205    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt" event={"ID":"c2822e0d-87da-4882-a4ed-acebabc7cc8b","Type":"ContainerDied","Data":"e3ceae8c9809c96c1c4edf12965e9a6f82bd420aa76ca83305d6e6665658bd4c"}
Oct 29 23:00:58.216144 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:58.216118    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"
Oct 29 23:00:58.216144 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:58.216135    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp" event={"ID":"11ae1a35-1d04-478b-be37-75c897b761c0","Type":"ContainerDied","Data":"bffa14c0bb97ee2ef4d3fa0b08ba73cf01ce0d9ca6e1e24a917d9611528dd3bc"}
Oct 29 23:00:58.216328 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:58.216184    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bffa14c0bb97ee2ef4d3fa0b08ba73cf01ce0d9ca6e1e24a917d9611528dd3bc"
Oct 29 23:00:58.969584 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:58.926208    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"
Oct 29 23:00:58.998784 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:58.991883    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"]
Oct 29 23:00:59.107127 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.107091    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2822e0d-87da-4882-a4ed-acebabc7cc8b-kube-api-access-r6s88\" (UniqueName: \"kubernetes.io/projected/c2822e0d-87da-4882-a4ed-acebabc7cc8b-kube-api-access-r6s88\") pod \"c2822e0d-87da-4882-a4ed-acebabc7cc8b\" (UID: \"c2822e0d-87da-4882-a4ed-acebabc7cc8b\") "
Oct 29 23:00:59.107386 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.107365    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2822e0d-87da-4882-a4ed-acebabc7cc8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2822e0d-87da-4882-a4ed-acebabc7cc8b-plugins-config-volume\") pod \"c2822e0d-87da-4882-a4ed-acebabc7cc8b\" (UID: \"c2822e0d-87da-4882-a4ed-acebabc7cc8b\") "
Oct 29 23:00:59.108595 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.108558    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2822e0d-87da-4882-a4ed-acebabc7cc8b-plugins-config-volume" pod "c2822e0d-87da-4882-a4ed-acebabc7cc8b" (UID: "c2822e0d-87da-4882-a4ed-acebabc7cc8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:59.108996 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.108974    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2822e0d-87da-4882-a4ed-acebabc7cc8b-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:59.114681 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.113736    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2822e0d-87da-4882-a4ed-acebabc7cc8b-kube-api-access-r6s88" pod "c2822e0d-87da-4882-a4ed-acebabc7cc8b" (UID: "c2822e0d-87da-4882-a4ed-acebabc7cc8b"). InnerVolumeSpecName "kube-api-access-r6s88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:59.117263 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.117233    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c933cb-hsbbl"]
Oct 29 23:00:59.122091 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.122070    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"
Oct 29 23:00:59.226979 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.226893    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370fbc49-ac99-41e3-9375-098433ebd4da-kube-api-access-854gr\" (UniqueName: \"kubernetes.io/projected/370fbc49-ac99-41e3-9375-098433ebd4da-kube-api-access-854gr\") pod \"370fbc49-ac99-41e3-9375-098433ebd4da\" (UID: \"370fbc49-ac99-41e3-9375-098433ebd4da\") "
Oct 29 23:00:59.227524 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.227491    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/370fbc49-ac99-41e3-9375-098433ebd4da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370fbc49-ac99-41e3-9375-098433ebd4da-plugins-config-volume\") pod \"370fbc49-ac99-41e3-9375-098433ebd4da\" (UID: \"370fbc49-ac99-41e3-9375-098433ebd4da\") "
Oct 29 23:00:59.228391 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.228289    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6s88\" (UniqueName: \"kubernetes.io/projected/c2822e0d-87da-4882-a4ed-acebabc7cc8b-kube-api-access-r6s88\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:59.230451 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.229288    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370fbc49-ac99-41e3-9375-098433ebd4da-plugins-config-volume" pod "370fbc49-ac99-41e3-9375-098433ebd4da" (UID: "370fbc49-ac99-41e3-9375-098433ebd4da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:59.234336 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:59.234291    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-conmon-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6a5323_5dab_421e_9c50_b28825aaa895.slice/crio-4b978c63699e8ab4ab4ea9f2ed09240271664b8f578990f0dc9414cf32a8e69c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ae42cd_1e69_44b6_b728_fb34e15e46c7.slice/crio-conmon-81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:00:59.235690 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:59.235640    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-conmon-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6a5323_5dab_421e_9c50_b28825aaa895.slice/crio-4b978c63699e8ab4ab4ea9f2ed09240271664b8f578990f0dc9414cf32a8e69c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ae42cd_1e69_44b6_b728_fb34e15e46c7.slice/crio-conmon-81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:00:59.280274 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.280183    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5" path="/var/lib/kubelet/pods/94e43c86-7cdb-4f7c-aaa3-48a33f03b2f5/volumes"
Oct 29 23:00:59.281025 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.280956    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370fbc49-ac99-41e3-9375-098433ebd4da-kube-api-access-854gr" pod "370fbc49-ac99-41e3-9375-098433ebd4da" (UID: "370fbc49-ac99-41e3-9375-098433ebd4da"). InnerVolumeSpecName "kube-api-access-854gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:59.281854 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.281371    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"
Oct 29 23:00:59.281854 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.281670    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt" event={"ID":"c2822e0d-87da-4882-a4ed-acebabc7cc8b","Type":"ContainerDied","Data":"71102278bcff5ecb77b95c71bc10a26dbc43d8029de86e426507e821a9260bd3"}
Oct 29 23:00:59.281854 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.281789    4559 scope.go:122] "RemoveContainer" containerID="e3ceae8c9809c96c1c4edf12965e9a6f82bd420aa76ca83305d6e6665658bd4c"
Oct 29 23:00:59.289597 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.289561    4559 generic.go:356] "Generic (PLEG): container finished" podID="370fbc49-ac99-41e3-9375-098433ebd4da" containerID="74c9029925c67c596c49882746be99c717f68fa59f9b666e66dbd98fa7f79f99" exitCode=0
Oct 29 23:00:59.289780 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.289753    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh" event={"ID":"370fbc49-ac99-41e3-9375-098433ebd4da","Type":"ContainerDied","Data":"74c9029925c67c596c49882746be99c717f68fa59f9b666e66dbd98fa7f79f99"}
Oct 29 23:00:59.290099 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.290065    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh" event={"ID":"370fbc49-ac99-41e3-9375-098433ebd4da","Type":"ContainerDied","Data":"536816625d71136de2e046f88bbae7093c1b0452528b140d61fd67a5e670fe13"}
Oct 29 23:00:59.290331 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.290317    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"
Oct 29 23:00:59.302850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.302784    4559 scope.go:122] "RemoveContainer" containerID="74c9029925c67c596c49882746be99c717f68fa59f9b666e66dbd98fa7f79f99"
Oct 29 23:00:59.315066 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.315043    4559 scope.go:122] "RemoveContainer" containerID="74c9029925c67c596c49882746be99c717f68fa59f9b666e66dbd98fa7f79f99"
Oct 29 23:00:59.315380 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:00:59.315358    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74c9029925c67c596c49882746be99c717f68fa59f9b666e66dbd98fa7f79f99\": container with ID starting with 74c9029925c67c596c49882746be99c717f68fa59f9b666e66dbd98fa7f79f99 not found: ID does not exist" containerID="74c9029925c67c596c49882746be99c717f68fa59f9b666e66dbd98fa7f79f99"
Oct 29 23:00:59.315448 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.315396    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74c9029925c67c596c49882746be99c717f68fa59f9b666e66dbd98fa7f79f99"} err="failed to get container status \"74c9029925c67c596c49882746be99c717f68fa59f9b666e66dbd98fa7f79f99\": rpc error: code = NotFound desc = could not find container \"74c9029925c67c596c49882746be99c717f68fa59f9b666e66dbd98fa7f79f99\": container with ID starting with 74c9029925c67c596c49882746be99c717f68fa59f9b666e66dbd98fa7f79f99 not found: ID does not exist"
Oct 29 23:00:59.330483 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.330450    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-854gr\" (UniqueName: \"kubernetes.io/projected/370fbc49-ac99-41e3-9375-098433ebd4da-kube-api-access-854gr\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:59.330483 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.330481    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370fbc49-ac99-41e3-9375-098433ebd4da-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:00:59.371841 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.371525    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"]
Oct 29 23:00:59.378136 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.378100    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt"]
Oct 29 23:00:59.400319 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.400281    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"]
Oct 29 23:00:59.404426 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.404396    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh"]
Oct 29 23:00:59.757837 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.756992    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-77657dc59-xt7mh" podUID="370fbc49-ac99-41e3-9375-098433ebd4da" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.242:50052\" within 1s: context deadline exceeded"
Oct 29 23:00:59.757837 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:00:59.757190    4559 logging.go:55] [core] [Channel #9641 SubChannel #9643]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.242:50052", ServerName: "10.134.0.242:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.242:50052: operation was canceled"
Oct 29 23:00:59.757837 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:00:59.757454    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-9f999d45d-9qwbt" podUID="c2822e0d-87da-4882-a4ed-acebabc7cc8b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.243:50052\" within 1s: context deadline exceeded"
Oct 29 23:00:59.757837 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:00:59.757597    4559 logging.go:55] [core] [Channel #9642 SubChannel #9644]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.243:50052", ServerName: "10.134.0.243:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.243:50052: operation was canceled"
Oct 29 23:01:01.231770 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:01.231730    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="370fbc49-ac99-41e3-9375-098433ebd4da" path="/var/lib/kubelet/pods/370fbc49-ac99-41e3-9375-098433ebd4da/volumes"
Oct 29 23:01:01.232417 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:01.232396    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2822e0d-87da-4882-a4ed-acebabc7cc8b" path="/var/lib/kubelet/pods/c2822e0d-87da-4882-a4ed-acebabc7cc8b/volumes"
Oct 29 23:01:01.588945 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:01.588842    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main" containerID="cri-o://b002d69b20b8a3b2f073462fcdc8d08ca0fac491ac93d3f4f477ef04243130f0" gracePeriod=45
Oct 29 23:01:01.915202 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:01.915167    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 23:01:01.973464 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:01:01.965399    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-conmon-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ae42cd_1e69_44b6_b728_fb34e15e46c7.slice/crio-conmon-81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6a5323_5dab_421e_9c50_b28825aaa895.slice/crio-4b978c63699e8ab4ab4ea9f2ed09240271664b8f578990f0dc9414cf32a8e69c.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:01:02.096892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.096853    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-dshm\") pod \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") "
Oct 29 23:01:02.097089 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.096916    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-home\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-home\") pod \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") "
Oct 29 23:01:02.097182 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.097161    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-kube-api-access-xgs49\" (UniqueName: \"kubernetes.io/projected/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-kube-api-access-xgs49\") pod \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") "
Oct 29 23:01:02.097240 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.097191    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-home" pod "53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" (UID: "53116ca8-8cee-4efb-8f3f-4ccb571ff7ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.097240 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.097214    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-model-cache\") pod \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") "
Oct 29 23:01:02.097393 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.097374    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-tls-certs\") pod \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") "
Oct 29 23:01:02.097464 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.097436    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-tmp-dir\") pod \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\" (UID: \"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca\") "
Oct 29 23:01:02.097526 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.097497    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-model-cache" pod "53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" (UID: "53116ca8-8cee-4efb-8f3f-4ccb571ff7ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.097770 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.097752    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-tmp-dir" pod "53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" (UID: "53116ca8-8cee-4efb-8f3f-4ccb571ff7ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.098171 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.098149    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.098171 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.098168    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.098375 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.098184    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.099423 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.099393    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-dshm" pod "53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" (UID: "53116ca8-8cee-4efb-8f3f-4ccb571ff7ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.099554 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.099507    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-kube-api-access-xgs49" pod "53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" (UID: "53116ca8-8cee-4efb-8f3f-4ccb571ff7ca"). InnerVolumeSpecName "kube-api-access-xgs49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:02.100144 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.100127    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-tls-certs" pod "53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" (UID: "53116ca8-8cee-4efb-8f3f-4ccb571ff7ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:02.200169 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.200129    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgs49\" (UniqueName: \"kubernetes.io/projected/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-kube-api-access-xgs49\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.200169 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.200164    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.200169 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.200172    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.325080 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.325039    4559 generic.go:356] "Generic (PLEG): container finished" podID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerID="b002d69b20b8a3b2f073462fcdc8d08ca0fac491ac93d3f4f477ef04243130f0" exitCode=0
Oct 29 23:01:02.325669 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.325101    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" event={"ID":"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca","Type":"ContainerDied","Data":"b002d69b20b8a3b2f073462fcdc8d08ca0fac491ac93d3f4f477ef04243130f0"}
Oct 29 23:01:02.325896 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.325721    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc" event={"ID":"53116ca8-8cee-4efb-8f3f-4ccb571ff7ca","Type":"ContainerDied","Data":"0ba376eae89d97bb53eee4ac541375d8c3f46eb662e34233537b917da8678770"}
Oct 29 23:01:02.325896 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.325759    4559 scope.go:122] "RemoveContainer" containerID="b002d69b20b8a3b2f073462fcdc8d08ca0fac491ac93d3f4f477ef04243130f0"
Oct 29 23:01:02.325896 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.325199    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"
Oct 29 23:01:02.342218 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.342195    4559 scope.go:122] "RemoveContainer" containerID="b002d69b20b8a3b2f073462fcdc8d08ca0fac491ac93d3f4f477ef04243130f0"
Oct 29 23:01:02.342595 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:01:02.342573    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b002d69b20b8a3b2f073462fcdc8d08ca0fac491ac93d3f4f477ef04243130f0\": container with ID starting with b002d69b20b8a3b2f073462fcdc8d08ca0fac491ac93d3f4f477ef04243130f0 not found: ID does not exist" containerID="b002d69b20b8a3b2f073462fcdc8d08ca0fac491ac93d3f4f477ef04243130f0"
Oct 29 23:01:02.342661 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.342606    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b002d69b20b8a3b2f073462fcdc8d08ca0fac491ac93d3f4f477ef04243130f0"} err="failed to get container status \"b002d69b20b8a3b2f073462fcdc8d08ca0fac491ac93d3f4f477ef04243130f0\": rpc error: code = NotFound desc = could not find container \"b002d69b20b8a3b2f073462fcdc8d08ca0fac491ac93d3f4f477ef04243130f0\": container with ID starting with b002d69b20b8a3b2f073462fcdc8d08ca0fac491ac93d3f4f477ef04243130f0 not found: ID does not exist"
Oct 29 23:01:02.385857 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.385820    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"]
Oct 29 23:01:02.392277 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:02.392242    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/premium-w3-43b37d-kserve-5bb66fbd9b-pzvbc"]
Oct 29 23:01:03.230657 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:03.230617    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" path="/var/lib/kubelet/pods/53116ca8-8cee-4efb-8f3f-4ccb571ff7ca/volumes"
Oct 29 23:01:04.418488 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:04.416829    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": read tcp 10.134.0.2:44030->10.134.0.241:15021: read: connection reset by peer" start-of-body=
Oct 29 23:01:04.418488 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:04.416898    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn" podUID="6f118086-bc70-4d6d-874a-12ebec737d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": read tcp 10.134.0.2:44030->10.134.0.241:15021: read: connection reset by peer"
Oct 29 23:01:05.394307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:05.394269    4559 generic.go:356] "Generic (PLEG): container finished" podID="6f118086-bc70-4d6d-874a-12ebec737d0d" containerID="9c9dd15fc4007ca1dc5545302ff96155c2433d00839fb6e349563d7097229e61" exitCode=0
Oct 29 23:01:05.394874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:05.394845    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn" event={"ID":"6f118086-bc70-4d6d-874a-12ebec737d0d","Type":"ContainerDied","Data":"9c9dd15fc4007ca1dc5545302ff96155c2433d00839fb6e349563d7097229e61"}
Oct 29 23:01:05.395635 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:05.395609    4559 scope.go:122] "RemoveContainer" containerID="9c9dd15fc4007ca1dc5545302ff96155c2433d00839fb6e349563d7097229e61"
Oct 29 23:01:05.415406 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:05.415350    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:01:05.415753 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:05.415717    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:01:05.416076 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:05.416054    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:01:05.665141 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:05.665113    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"]
Oct 29 23:01:05.687358 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:05.687323    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"]
Oct 29 23:01:05.984985 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:05.984885    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"]
Oct 29 23:01:05.985533 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:05.985500    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84" podUID="b905ef8e-61e8-4693-ae47-1b3940e87b0d" containerName="istio-proxy" containerID="cri-o://4c77f80dafa1521bffc45f05ccd21021095594b8cb7e09b51fb036430c2a7822" gracePeriod=30
Oct 29 23:01:06.405242 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.405209    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn" event={"ID":"6f118086-bc70-4d6d-874a-12ebec737d0d","Type":"ContainerStarted","Data":"59c509d1c569f337f550c077bbe52831b0e237ddf1f21a3c5a355da8444f2732"}
Oct 29 23:01:06.428306 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.428266    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"]
Oct 29 23:01:06.429188 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.429157    4559 cpu_manager.go:414] "containerMap: removing container" podUID="370fbc49-ac99-41e3-9375-098433ebd4da" containerName="payload-pre-processing"
Oct 29 23:01:06.429188 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.429186    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370fbc49-ac99-41e3-9375-098433ebd4da" containerName="payload-pre-processing"
Oct 29 23:01:06.429460 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.429232    4559 cpu_manager.go:414] "containerMap: removing container" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main"
Oct 29 23:01:06.429460 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.429241    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main"
Oct 29 23:01:06.429460 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.429255    4559 cpu_manager.go:414] "containerMap: removing container" podUID="11ae1a35-1d04-478b-be37-75c897b761c0" containerName="revoke-keys"
Oct 29 23:01:06.429460 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.429262    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ae1a35-1d04-478b-be37-75c897b761c0" containerName="revoke-keys"
Oct 29 23:01:06.429460 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.429311    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c2822e0d-87da-4882-a4ed-acebabc7cc8b" containerName="payload-processing"
Oct 29 23:01:06.429460 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.429318    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2822e0d-87da-4882-a4ed-acebabc7cc8b" containerName="payload-processing"
Oct 29 23:01:06.429788 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.429499    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="370fbc49-ac99-41e3-9375-098433ebd4da" containerName="payload-pre-processing"
Oct 29 23:01:06.429788 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.429519    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="53116ca8-8cee-4efb-8f3f-4ccb571ff7ca" containerName="main"
Oct 29 23:01:06.429788 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.429530    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ae1a35-1d04-478b-be37-75c897b761c0" containerName="revoke-keys"
Oct 29 23:01:06.429788 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.429546    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2822e0d-87da-4882-a4ed-acebabc7cc8b" containerName="payload-processing"
Oct 29 23:01:06.449661 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.443150    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"
Oct 29 23:01:06.481005 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.469361    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:01:06.481005 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.469531    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:01:06.507527 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.505802    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64eb694-3958-4dec-a8cc-beaf163d64d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x\" (UID: \"d64eb694-3958-4dec-a8cc-beaf163d64d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"
Oct 29 23:01:06.507527 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.505934    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64eb694-3958-4dec-a8cc-beaf163d64d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x\" (UID: \"d64eb694-3958-4dec-a8cc-beaf163d64d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"
Oct 29 23:01:06.507527 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.506169    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp8vd\" (UniqueName: \"kubernetes.io/projected/d64eb694-3958-4dec-a8cc-beaf163d64d4-kube-api-access-lp8vd\") pod \"maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x\" (UID: \"d64eb694-3958-4dec-a8cc-beaf163d64d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"
Oct 29 23:01:06.596360 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.596321    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"]
Oct 29 23:01:06.627836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.616850    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp8vd\" (UniqueName: \"kubernetes.io/projected/d64eb694-3958-4dec-a8cc-beaf163d64d4-kube-api-access-lp8vd\") pod \"maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x\" (UID: \"d64eb694-3958-4dec-a8cc-beaf163d64d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"
Oct 29 23:01:06.627836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.617048    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64eb694-3958-4dec-a8cc-beaf163d64d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x\" (UID: \"d64eb694-3958-4dec-a8cc-beaf163d64d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"
Oct 29 23:01:06.627836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.618589    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64eb694-3958-4dec-a8cc-beaf163d64d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x\" (UID: \"d64eb694-3958-4dec-a8cc-beaf163d64d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"
Oct 29 23:01:06.635906 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.635868    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64eb694-3958-4dec-a8cc-beaf163d64d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x\" (UID: \"d64eb694-3958-4dec-a8cc-beaf163d64d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"
Oct 29 23:01:06.672928 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.670316    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64eb694-3958-4dec-a8cc-beaf163d64d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x\" (UID: \"d64eb694-3958-4dec-a8cc-beaf163d64d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"
Oct 29 23:01:06.672928 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.671187    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp8vd\" (UniqueName: \"kubernetes.io/projected/d64eb694-3958-4dec-a8cc-beaf163d64d4-kube-api-access-lp8vd\") pod \"maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x\" (UID: \"d64eb694-3958-4dec-a8cc-beaf163d64d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"
Oct 29 23:01:06.672928 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.672588    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"]
Oct 29 23:01:06.686359 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.682920    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"
Oct 29 23:01:06.696476 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.696450    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"]
Oct 29 23:01:06.773835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.769663    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9l75\" (UniqueName: \"kubernetes.io/projected/de7c3068-21b6-4ae0-9f35-c60e986538c7-kube-api-access-m9l75\") pod \"maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv\" (UID: \"de7c3068-21b6-4ae0-9f35-c60e986538c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"
Oct 29 23:01:06.773835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.769752    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7c3068-21b6-4ae0-9f35-c60e986538c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv\" (UID: \"de7c3068-21b6-4ae0-9f35-c60e986538c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"
Oct 29 23:01:06.773835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.769996    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7c3068-21b6-4ae0-9f35-c60e986538c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv\" (UID: \"de7c3068-21b6-4ae0-9f35-c60e986538c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"
Oct 29 23:01:06.779838 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.777521    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 23:01:06.782992 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.782951    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Oct 29 23:01:06.783183 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.783016    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn" podUID="6f118086-bc70-4d6d-874a-12ebec737d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Oct 29 23:01:06.871754 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.871700    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7c3068-21b6-4ae0-9f35-c60e986538c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv\" (UID: \"de7c3068-21b6-4ae0-9f35-c60e986538c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"
Oct 29 23:01:06.872015 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.871933    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9l75\" (UniqueName: \"kubernetes.io/projected/de7c3068-21b6-4ae0-9f35-c60e986538c7-kube-api-access-m9l75\") pod \"maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv\" (UID: \"de7c3068-21b6-4ae0-9f35-c60e986538c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"
Oct 29 23:01:06.872015 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.871963    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7c3068-21b6-4ae0-9f35-c60e986538c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv\" (UID: \"de7c3068-21b6-4ae0-9f35-c60e986538c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"
Oct 29 23:01:06.872496 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.872471    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7c3068-21b6-4ae0-9f35-c60e986538c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv\" (UID: \"de7c3068-21b6-4ae0-9f35-c60e986538c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"
Oct 29 23:01:06.874608 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.874583    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7c3068-21b6-4ae0-9f35-c60e986538c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv\" (UID: \"de7c3068-21b6-4ae0-9f35-c60e986538c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"
Oct 29 23:01:06.892338 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.892306    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9l75\" (UniqueName: \"kubernetes.io/projected/de7c3068-21b6-4ae0-9f35-c60e986538c7-kube-api-access-m9l75\") pod \"maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv\" (UID: \"de7c3068-21b6-4ae0-9f35-c60e986538c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"
Oct 29 23:01:06.897194 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:06.897170    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"
Oct 29 23:01:07.013974 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:07.013932    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"
Oct 29 23:01:07.214175 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:01:07.214127    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6a5323_5dab_421e_9c50_b28825aaa895.slice/crio-4b978c63699e8ab4ab4ea9f2ed09240271664b8f578990f0dc9414cf32a8e69c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ae42cd_1e69_44b6_b728_fb34e15e46c7.slice/crio-conmon-81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-conmon-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:01:07.294307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:07.294268    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"]
Oct 29 23:01:07.416588 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:07.416548    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x" event={"ID":"d64eb694-3958-4dec-a8cc-beaf163d64d4","Type":"ContainerStarted","Data":"37f360bfe04612c81b4b626dbdbfa72d359feb84c2776b9146d957625eaad331"}
Oct 29 23:01:07.419426 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:07.419400    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"]
Oct 29 23:01:07.420404 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:01:07.420376    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7c3068_21b6_4ae0_9f35_c60e986538c7.slice/crio-1d73caf791a50beb01096da5cce7625e86f416281a5b78ad2c91f561d0d8357c WatchSource:0}: Error finding container 1d73caf791a50beb01096da5cce7625e86f416281a5b78ad2c91f561d0d8357c: Status 404 returned error can't find the container with id 1d73caf791a50beb01096da5cce7625e86f416281a5b78ad2c91f561d0d8357c
Oct 29 23:01:07.776008 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:07.775975    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Oct 29 23:01:07.776476 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:07.776046    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn" podUID="6f118086-bc70-4d6d-874a-12ebec737d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Oct 29 23:01:08.427306 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:08.427271    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x" event={"ID":"d64eb694-3958-4dec-a8cc-beaf163d64d4","Type":"ContainerStarted","Data":"2b5173ea0b5b6c45f93a7f73ca2e6036ead0ac1e23314330aeca408528d653c7"}
Oct 29 23:01:08.429065 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:08.429038    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv" event={"ID":"de7c3068-21b6-4ae0-9f35-c60e986538c7","Type":"ContainerStarted","Data":"edfd79d429447c34b5965f4052369dbf18b364dc1b68db967eaa4852606ae7fc"}
Oct 29 23:01:08.429208 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:08.429085    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv" event={"ID":"de7c3068-21b6-4ae0-9f35-c60e986538c7","Type":"ContainerStarted","Data":"1d73caf791a50beb01096da5cce7625e86f416281a5b78ad2c91f561d0d8357c"}
Oct 29 23:01:08.473013 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:08.472953    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x" podStartSLOduration=2.472931001 podStartE2EDuration="2.472931001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:08.469645076 +0000 UTC m=+2807.381088630" watchObservedRunningTime="2026-10-29 23:01:08.472931001 +0000 UTC m=+2807.384374558"
Oct 29 23:01:08.513529 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:08.513472    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv" podStartSLOduration=2.51344848 podStartE2EDuration="2.51344848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:08.510437381 +0000 UTC m=+2807.421880948" watchObservedRunningTime="2026-10-29 23:01:08.51344848 +0000 UTC m=+2807.424892037"
Oct 29 23:01:08.775566 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:08.775524    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Oct 29 23:01:08.775765 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:08.775618    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn" podUID="6f118086-bc70-4d6d-874a-12ebec737d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Oct 29 23:01:09.479833 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:09.479400    4559 generic.go:356] "Generic (PLEG): container finished" podID="de7c3068-21b6-4ae0-9f35-c60e986538c7" containerID="edfd79d429447c34b5965f4052369dbf18b364dc1b68db967eaa4852606ae7fc" exitCode=0
Oct 29 23:01:09.479833 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:09.479729    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv" event={"ID":"de7c3068-21b6-4ae0-9f35-c60e986538c7","Type":"ContainerDied","Data":"edfd79d429447c34b5965f4052369dbf18b364dc1b68db967eaa4852606ae7fc"}
Oct 29 23:01:09.485848 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:09.482597    4559 generic.go:356] "Generic (PLEG): container finished" podID="d64eb694-3958-4dec-a8cc-beaf163d64d4" containerID="2b5173ea0b5b6c45f93a7f73ca2e6036ead0ac1e23314330aeca408528d653c7" exitCode=0
Oct 29 23:01:09.485848 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:09.482677    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x" event={"ID":"d64eb694-3958-4dec-a8cc-beaf163d64d4","Type":"ContainerDied","Data":"2b5173ea0b5b6c45f93a7f73ca2e6036ead0ac1e23314330aeca408528d653c7"}
Oct 29 23:01:09.793122 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:09.792791    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Oct 29 23:01:09.796269 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:09.794849    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn" podUID="6f118086-bc70-4d6d-874a-12ebec737d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Oct 29 23:01:09.968916 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:09.968871    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"]
Oct 29 23:01:10.006025 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.005993    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"]
Oct 29 23:01:10.006368 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.006351    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"
Oct 29 23:01:10.053899 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.037249    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-80248b-dockercfg-6tw4j\""
Oct 29 23:01:10.109961 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.054395    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-80248b\""
Oct 29 23:01:10.168852 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.162524    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fc466b-5f14-4f21-bba8-d8be8f30323f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml\" (UID: \"a0fc466b-5f14-4f21-bba8-d8be8f30323f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"
Oct 29 23:01:10.168852 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.162718    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxd6k\" (UniqueName: \"kubernetes.io/projected/a0fc466b-5f14-4f21-bba8-d8be8f30323f-kube-api-access-bxd6k\") pod \"payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml\" (UID: \"a0fc466b-5f14-4f21-bba8-d8be8f30323f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"
Oct 29 23:01:10.186478 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.186176    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"]
Oct 29 23:01:10.191114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.191082    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"]
Oct 29 23:01:10.191307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.191198    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"
Oct 29 23:01:10.268070 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.268008    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h7qk\" (UniqueName: \"kubernetes.io/projected/db95228d-b0d3-41df-966d-770f4f338a72-kube-api-access-5h7qk\") pod \"payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6\" (UID: \"db95228d-b0d3-41df-966d-770f4f338a72\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"
Oct 29 23:01:10.270964 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.270924    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxd6k\" (UniqueName: \"kubernetes.io/projected/a0fc466b-5f14-4f21-bba8-d8be8f30323f-kube-api-access-bxd6k\") pod \"payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml\" (UID: \"a0fc466b-5f14-4f21-bba8-d8be8f30323f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"
Oct 29 23:01:10.271803 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.271774    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db95228d-b0d3-41df-966d-770f4f338a72-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6\" (UID: \"db95228d-b0d3-41df-966d-770f4f338a72\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"
Oct 29 23:01:10.272205 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.272182    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fc466b-5f14-4f21-bba8-d8be8f30323f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml\" (UID: \"a0fc466b-5f14-4f21-bba8-d8be8f30323f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"
Oct 29 23:01:10.273727 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.273701    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fc466b-5f14-4f21-bba8-d8be8f30323f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml\" (UID: \"a0fc466b-5f14-4f21-bba8-d8be8f30323f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"
Oct 29 23:01:10.297089 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.292241    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxd6k\" (UniqueName: \"kubernetes.io/projected/a0fc466b-5f14-4f21-bba8-d8be8f30323f-kube-api-access-bxd6k\") pod \"payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml\" (UID: \"a0fc466b-5f14-4f21-bba8-d8be8f30323f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"
Oct 29 23:01:10.374167 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.374058    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h7qk\" (UniqueName: \"kubernetes.io/projected/db95228d-b0d3-41df-966d-770f4f338a72-kube-api-access-5h7qk\") pod \"payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6\" (UID: \"db95228d-b0d3-41df-966d-770f4f338a72\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"
Oct 29 23:01:10.374380 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.374269    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db95228d-b0d3-41df-966d-770f4f338a72-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6\" (UID: \"db95228d-b0d3-41df-966d-770f4f338a72\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"
Oct 29 23:01:10.375287 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.375259    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db95228d-b0d3-41df-966d-770f4f338a72-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6\" (UID: \"db95228d-b0d3-41df-966d-770f4f338a72\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"
Oct 29 23:01:10.394383 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.394342    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h7qk\" (UniqueName: \"kubernetes.io/projected/db95228d-b0d3-41df-966d-770f4f338a72-kube-api-access-5h7qk\") pod \"payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6\" (UID: \"db95228d-b0d3-41df-966d-770f4f338a72\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"
Oct 29 23:01:10.490079 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.490046    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"
Oct 29 23:01:10.515143 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.514896    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"
Oct 29 23:01:10.675841 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.668903    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k" podUID="0400c09d-e610-4055-b068-c00c6f156c9c" containerName="payload-processing" containerID="cri-o://6af1c113c396b1f24cbf5fa23c33de79b401cb398f84d0eac0dca7f990b8d8c0" gracePeriod=25
Oct 29 23:01:10.717884 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.701470    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9" podUID="6b93002e-a2d9-4bd4-9bb3-266576873ebb" containerName="payload-pre-processing" containerID="cri-o://429f204bcaa63c9fe7ee321ba7a3a0f45d521631f2c01610992cfe9d69122593" gracePeriod=25
Oct 29 23:01:10.776016 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.775510    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Oct 29 23:01:10.776016 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:10.775585    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn" podUID="6f118086-bc70-4d6d-874a-12ebec737d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Oct 29 23:01:11.012157 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.006466    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"
Oct 29 23:01:11.016103 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.013306    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"
Oct 29 23:01:11.129430 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.128560    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d64eb694-3958-4dec-a8cc-beaf163d64d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64eb694-3958-4dec-a8cc-beaf163d64d4-maas-api-openshift-ca\") pod \"d64eb694-3958-4dec-a8cc-beaf163d64d4\" (UID: \"d64eb694-3958-4dec-a8cc-beaf163d64d4\") "
Oct 29 23:01:11.129430 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.128625    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d64eb694-3958-4dec-a8cc-beaf163d64d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64eb694-3958-4dec-a8cc-beaf163d64d4-maas-api-xks-ca\") pod \"d64eb694-3958-4dec-a8cc-beaf163d64d4\" (UID: \"d64eb694-3958-4dec-a8cc-beaf163d64d4\") "
Oct 29 23:01:11.129430 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.128691    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7c3068-21b6-4ae0-9f35-c60e986538c7-kube-api-access-m9l75\" (UniqueName: \"kubernetes.io/projected/de7c3068-21b6-4ae0-9f35-c60e986538c7-kube-api-access-m9l75\") pod \"de7c3068-21b6-4ae0-9f35-c60e986538c7\" (UID: \"de7c3068-21b6-4ae0-9f35-c60e986538c7\") "
Oct 29 23:01:11.129430 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.128742    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64eb694-3958-4dec-a8cc-beaf163d64d4-kube-api-access-lp8vd\" (UniqueName: \"kubernetes.io/projected/d64eb694-3958-4dec-a8cc-beaf163d64d4-kube-api-access-lp8vd\") pod \"d64eb694-3958-4dec-a8cc-beaf163d64d4\" (UID: \"d64eb694-3958-4dec-a8cc-beaf163d64d4\") "
Oct 29 23:01:11.129430 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.128891    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7c3068-21b6-4ae0-9f35-c60e986538c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7c3068-21b6-4ae0-9f35-c60e986538c7-maas-api-openshift-ca\") pod \"de7c3068-21b6-4ae0-9f35-c60e986538c7\" (UID: \"de7c3068-21b6-4ae0-9f35-c60e986538c7\") "
Oct 29 23:01:11.129430 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.128913    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de7c3068-21b6-4ae0-9f35-c60e986538c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7c3068-21b6-4ae0-9f35-c60e986538c7-maas-api-xks-ca\") pod \"de7c3068-21b6-4ae0-9f35-c60e986538c7\" (UID: \"de7c3068-21b6-4ae0-9f35-c60e986538c7\") "
Oct 29 23:01:11.130219 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.129961    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64eb694-3958-4dec-a8cc-beaf163d64d4-maas-api-openshift-ca" pod "d64eb694-3958-4dec-a8cc-beaf163d64d4" (UID: "d64eb694-3958-4dec-a8cc-beaf163d64d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:11.130416 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.130387    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7c3068-21b6-4ae0-9f35-c60e986538c7-maas-api-openshift-ca" pod "de7c3068-21b6-4ae0-9f35-c60e986538c7" (UID: "de7c3068-21b6-4ae0-9f35-c60e986538c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:11.179658 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.179502    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de7c3068-21b6-4ae0-9f35-c60e986538c7-maas-api-xks-ca" pod "de7c3068-21b6-4ae0-9f35-c60e986538c7" (UID: "de7c3068-21b6-4ae0-9f35-c60e986538c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:11.202360 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.187199    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7c3068-21b6-4ae0-9f35-c60e986538c7-kube-api-access-m9l75" pod "de7c3068-21b6-4ae0-9f35-c60e986538c7" (UID: "de7c3068-21b6-4ae0-9f35-c60e986538c7"). InnerVolumeSpecName "kube-api-access-m9l75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.202360 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.194643    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d64eb694-3958-4dec-a8cc-beaf163d64d4-maas-api-xks-ca" pod "d64eb694-3958-4dec-a8cc-beaf163d64d4" (UID: "d64eb694-3958-4dec-a8cc-beaf163d64d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:11.216502 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.215104    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64eb694-3958-4dec-a8cc-beaf163d64d4-kube-api-access-lp8vd" pod "d64eb694-3958-4dec-a8cc-beaf163d64d4" (UID: "d64eb694-3958-4dec-a8cc-beaf163d64d4"). InnerVolumeSpecName "kube-api-access-lp8vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.274001 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.273959    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64eb694-3958-4dec-a8cc-beaf163d64d4-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.274278 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.274257    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64eb694-3958-4dec-a8cc-beaf163d64d4-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.274417 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.274402    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9l75\" (UniqueName: \"kubernetes.io/projected/de7c3068-21b6-4ae0-9f35-c60e986538c7-kube-api-access-m9l75\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.274521 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.274508    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp8vd\" (UniqueName: \"kubernetes.io/projected/d64eb694-3958-4dec-a8cc-beaf163d64d4-kube-api-access-lp8vd\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.274613 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.274601    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7c3068-21b6-4ae0-9f35-c60e986538c7-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.274703 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.274692    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7c3068-21b6-4ae0-9f35-c60e986538c7-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.304560 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.303857    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"]
Oct 29 23:01:11.403352 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.403318    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"]
Oct 29 23:01:11.409507 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:01:11.409478    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0fc466b_5f14_4f21_bba8_d8be8f30323f.slice/crio-61712457b1e2312a688dee9f3f0f57330302fe5843bd3ccf22e7759db453fe36 WatchSource:0}: Error finding container 61712457b1e2312a688dee9f3f0f57330302fe5843bd3ccf22e7759db453fe36: Status 404 returned error can't find the container with id 61712457b1e2312a688dee9f3f0f57330302fe5843bd3ccf22e7759db453fe36
Oct 29 23:01:11.412596 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.412558    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"
Oct 29 23:01:11.456771 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.445691    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"
Oct 29 23:01:11.490860 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.489591    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b93002e-a2d9-4bd4-9bb3-266576873ebb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b93002e-a2d9-4bd4-9bb3-266576873ebb-plugins-config-volume\") pod \"6b93002e-a2d9-4bd4-9bb3-266576873ebb\" (UID: \"6b93002e-a2d9-4bd4-9bb3-266576873ebb\") "
Oct 29 23:01:11.490860 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.489753    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b93002e-a2d9-4bd4-9bb3-266576873ebb-kube-api-access-gnbh9\" (UniqueName: \"kubernetes.io/projected/6b93002e-a2d9-4bd4-9bb3-266576873ebb-kube-api-access-gnbh9\") pod \"6b93002e-a2d9-4bd4-9bb3-266576873ebb\" (UID: \"6b93002e-a2d9-4bd4-9bb3-266576873ebb\") "
Oct 29 23:01:11.492688 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.492649    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b93002e-a2d9-4bd4-9bb3-266576873ebb-plugins-config-volume" pod "6b93002e-a2d9-4bd4-9bb3-266576873ebb" (UID: "6b93002e-a2d9-4bd4-9bb3-266576873ebb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:11.492844 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.492714    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"]
Oct 29 23:01:11.504093 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.504054    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"]
Oct 29 23:01:11.509101 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.509058    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b93002e-a2d9-4bd4-9bb3-266576873ebb-kube-api-access-gnbh9" pod "6b93002e-a2d9-4bd4-9bb3-266576873ebb" (UID: "6b93002e-a2d9-4bd4-9bb3-266576873ebb"). InnerVolumeSpecName "kube-api-access-gnbh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.575597 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.564393    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x" event={"ID":"d64eb694-3958-4dec-a8cc-beaf163d64d4","Type":"ContainerDied","Data":"37f360bfe04612c81b4b626dbdbfa72d359feb84c2776b9146d957625eaad331"}
Oct 29 23:01:11.575597 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.564464    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37f360bfe04612c81b4b626dbdbfa72d359feb84c2776b9146d957625eaad331"
Oct 29 23:01:11.575597 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.564535    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"
Oct 29 23:01:11.576435 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.576376    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv" event={"ID":"de7c3068-21b6-4ae0-9f35-c60e986538c7","Type":"ContainerDied","Data":"1d73caf791a50beb01096da5cce7625e86f416281a5b78ad2c91f561d0d8357c"}
Oct 29 23:01:11.576754 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.576676    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d73caf791a50beb01096da5cce7625e86f416281a5b78ad2c91f561d0d8357c"
Oct 29 23:01:11.578386 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.577026    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"
Oct 29 23:01:11.581593 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.581492    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6" event={"ID":"db95228d-b0d3-41df-966d-770f4f338a72","Type":"ContainerStarted","Data":"883e9c661583d603ec99e9077172da786ec2f3dd71a2540aaab1b4f9efbd06a9"}
Oct 29 23:01:11.598970 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.598718    4559 generic.go:356] "Generic (PLEG): container finished" podID="6b93002e-a2d9-4bd4-9bb3-266576873ebb" containerID="429f204bcaa63c9fe7ee321ba7a3a0f45d521631f2c01610992cfe9d69122593" exitCode=0
Oct 29 23:01:11.600461 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.599395    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9" event={"ID":"6b93002e-a2d9-4bd4-9bb3-266576873ebb","Type":"ContainerDied","Data":"429f204bcaa63c9fe7ee321ba7a3a0f45d521631f2c01610992cfe9d69122593"}
Oct 29 23:01:11.600461 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.599479    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9" event={"ID":"6b93002e-a2d9-4bd4-9bb3-266576873ebb","Type":"ContainerDied","Data":"d85d61977c1fb1e32d59b168b387f48a0e0a12119966fe9f40e7e4cc43966cc4"}
Oct 29 23:01:11.600461 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.599523    4559 scope.go:122] "RemoveContainer" containerID="429f204bcaa63c9fe7ee321ba7a3a0f45d521631f2c01610992cfe9d69122593"
Oct 29 23:01:11.600461 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.599738    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"
Oct 29 23:01:11.607131 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.605453    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0400c09d-e610-4055-b068-c00c6f156c9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0400c09d-e610-4055-b068-c00c6f156c9c-plugins-config-volume\") pod \"0400c09d-e610-4055-b068-c00c6f156c9c\" (UID: \"0400c09d-e610-4055-b068-c00c6f156c9c\") "
Oct 29 23:01:11.607131 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.605628    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0400c09d-e610-4055-b068-c00c6f156c9c-kube-api-access-mbwr7\" (UniqueName: \"kubernetes.io/projected/0400c09d-e610-4055-b068-c00c6f156c9c-kube-api-access-mbwr7\") pod \"0400c09d-e610-4055-b068-c00c6f156c9c\" (UID: \"0400c09d-e610-4055-b068-c00c6f156c9c\") "
Oct 29 23:01:11.607131 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.606388    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b93002e-a2d9-4bd4-9bb3-266576873ebb-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.607131 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.606415    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnbh9\" (UniqueName: \"kubernetes.io/projected/6b93002e-a2d9-4bd4-9bb3-266576873ebb-kube-api-access-gnbh9\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.614876 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.614471    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0400c09d-e610-4055-b068-c00c6f156c9c-plugins-config-volume" pod "0400c09d-e610-4055-b068-c00c6f156c9c" (UID: "0400c09d-e610-4055-b068-c00c6f156c9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:11.616248 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.615741    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0400c09d-e610-4055-b068-c00c6f156c9c-kube-api-access-mbwr7" pod "0400c09d-e610-4055-b068-c00c6f156c9c" (UID: "0400c09d-e610-4055-b068-c00c6f156c9c"). InnerVolumeSpecName "kube-api-access-mbwr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.621114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.620946    4559 generic.go:356] "Generic (PLEG): container finished" podID="b905ef8e-61e8-4693-ae47-1b3940e87b0d" containerID="4c77f80dafa1521bffc45f05ccd21021095594b8cb7e09b51fb036430c2a7822" exitCode=0
Oct 29 23:01:11.622681 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.621323    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84" event={"ID":"b905ef8e-61e8-4693-ae47-1b3940e87b0d","Type":"ContainerDied","Data":"4c77f80dafa1521bffc45f05ccd21021095594b8cb7e09b51fb036430c2a7822"}
Oct 29 23:01:11.625985 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.625754    4559 generic.go:356] "Generic (PLEG): container finished" podID="0400c09d-e610-4055-b068-c00c6f156c9c" containerID="6af1c113c396b1f24cbf5fa23c33de79b401cb398f84d0eac0dca7f990b8d8c0" exitCode=0
Oct 29 23:01:11.625985 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.625869    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k" event={"ID":"0400c09d-e610-4055-b068-c00c6f156c9c","Type":"ContainerDied","Data":"6af1c113c396b1f24cbf5fa23c33de79b401cb398f84d0eac0dca7f990b8d8c0"}
Oct 29 23:01:11.625985 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.625921    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k" event={"ID":"0400c09d-e610-4055-b068-c00c6f156c9c","Type":"ContainerDied","Data":"aecfe230c2d7645b1866dd63007572822f1456b422f4e7528e885265bb3e6cf6"}
Oct 29 23:01:11.626196 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.625995    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"
Oct 29 23:01:11.630344 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.630312    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml" event={"ID":"a0fc466b-5f14-4f21-bba8-d8be8f30323f","Type":"ContainerStarted","Data":"61712457b1e2312a688dee9f3f0f57330302fe5843bd3ccf22e7759db453fe36"}
Oct 29 23:01:11.676124 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.676087    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 23:01:11.680334 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.680301    4559 scope.go:122] "RemoveContainer" containerID="429f204bcaa63c9fe7ee321ba7a3a0f45d521631f2c01610992cfe9d69122593"
Oct 29 23:01:11.680886 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:01:11.680744    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"429f204bcaa63c9fe7ee321ba7a3a0f45d521631f2c01610992cfe9d69122593\": container with ID starting with 429f204bcaa63c9fe7ee321ba7a3a0f45d521631f2c01610992cfe9d69122593 not found: ID does not exist" containerID="429f204bcaa63c9fe7ee321ba7a3a0f45d521631f2c01610992cfe9d69122593"
Oct 29 23:01:11.680886 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.680776    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429f204bcaa63c9fe7ee321ba7a3a0f45d521631f2c01610992cfe9d69122593"} err="failed to get container status \"429f204bcaa63c9fe7ee321ba7a3a0f45d521631f2c01610992cfe9d69122593\": rpc error: code = NotFound desc = could not find container \"429f204bcaa63c9fe7ee321ba7a3a0f45d521631f2c01610992cfe9d69122593\": container with ID starting with 429f204bcaa63c9fe7ee321ba7a3a0f45d521631f2c01610992cfe9d69122593 not found: ID does not exist"
Oct 29 23:01:11.680886 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.680802    4559 scope.go:122] "RemoveContainer" containerID="6345299c7471ecfc26af9d2495cc7bcddb76905dcdefb020bb29f8558b74e683"
Oct 29 23:01:11.697263 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.696963    4559 scope.go:122] "RemoveContainer" containerID="6af1c113c396b1f24cbf5fa23c33de79b401cb398f84d0eac0dca7f990b8d8c0"
Oct 29 23:01:11.708097 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.708058    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbwr7\" (UniqueName: \"kubernetes.io/projected/0400c09d-e610-4055-b068-c00c6f156c9c-kube-api-access-mbwr7\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.708097 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.708100    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0400c09d-e610-4055-b068-c00c6f156c9c-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.711527 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.711490    4559 scope.go:122] "RemoveContainer" containerID="6af1c113c396b1f24cbf5fa23c33de79b401cb398f84d0eac0dca7f990b8d8c0"
Oct 29 23:01:11.711901 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:01:11.711873    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af1c113c396b1f24cbf5fa23c33de79b401cb398f84d0eac0dca7f990b8d8c0\": container with ID starting with 6af1c113c396b1f24cbf5fa23c33de79b401cb398f84d0eac0dca7f990b8d8c0 not found: ID does not exist" containerID="6af1c113c396b1f24cbf5fa23c33de79b401cb398f84d0eac0dca7f990b8d8c0"
Oct 29 23:01:11.712039 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.711913    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af1c113c396b1f24cbf5fa23c33de79b401cb398f84d0eac0dca7f990b8d8c0"} err="failed to get container status \"6af1c113c396b1f24cbf5fa23c33de79b401cb398f84d0eac0dca7f990b8d8c0\": rpc error: code = NotFound desc = could not find container \"6af1c113c396b1f24cbf5fa23c33de79b401cb398f84d0eac0dca7f990b8d8c0\": container with ID starting with 6af1c113c396b1f24cbf5fa23c33de79b401cb398f84d0eac0dca7f990b8d8c0 not found: ID does not exist"
Oct 29 23:01:11.775369 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.775338    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Oct 29 23:01:11.775656 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.775627    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn" podUID="6f118086-bc70-4d6d-874a-12ebec737d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Oct 29 23:01:11.810724 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.810194    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-podinfo\") pod \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") "
Oct 29 23:01:11.810724 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.810668    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-workload-socket\") pod \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") "
Oct 29 23:01:11.811296 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.810739    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-credential-socket\") pod \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") "
Oct 29 23:01:11.811296 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.810915    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-data\") pod \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") "
Oct 29 23:01:11.811296 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.810970    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istiod-ca-cert\") pod \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") "
Oct 29 23:01:11.811296 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.811047    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-workload-certs\") pod \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") "
Oct 29 23:01:11.811296 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.811106    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-token\" (UniqueName: \"kubernetes.io/projected/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-token\") pod \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") "
Oct 29 23:01:11.811296 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.811176    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-envoy\") pod \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") "
Oct 29 23:01:11.811296 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.811209    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b905ef8e-61e8-4693-ae47-1b3940e87b0d-kube-api-access-r4pkb\" (UniqueName: \"kubernetes.io/projected/b905ef8e-61e8-4693-ae47-1b3940e87b0d-kube-api-access-r4pkb\") pod \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\" (UID: \"b905ef8e-61e8-4693-ae47-1b3940e87b0d\") "
Oct 29 23:01:11.812439 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.811779    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istiod-ca-cert" pod "b905ef8e-61e8-4693-ae47-1b3940e87b0d" (UID: "b905ef8e-61e8-4693-ae47-1b3940e87b0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:11.812439 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.812053    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.812439 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.812126    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-data" pod "b905ef8e-61e8-4693-ae47-1b3940e87b0d" (UID: "b905ef8e-61e8-4693-ae47-1b3940e87b0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.812439 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.812355    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-workload-certs" pod "b905ef8e-61e8-4693-ae47-1b3940e87b0d" (UID: "b905ef8e-61e8-4693-ae47-1b3940e87b0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.812678 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.812662    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-credential-socket" pod "b905ef8e-61e8-4693-ae47-1b3940e87b0d" (UID: "b905ef8e-61e8-4693-ae47-1b3940e87b0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.813021 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.812938    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-workload-socket" pod "b905ef8e-61e8-4693-ae47-1b3940e87b0d" (UID: "b905ef8e-61e8-4693-ae47-1b3940e87b0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.815861 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.815388    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b905ef8e-61e8-4693-ae47-1b3940e87b0d-kube-api-access-r4pkb" pod "b905ef8e-61e8-4693-ae47-1b3940e87b0d" (UID: "b905ef8e-61e8-4693-ae47-1b3940e87b0d"). InnerVolumeSpecName "kube-api-access-r4pkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.815861 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.815545    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-podinfo" pod "b905ef8e-61e8-4693-ae47-1b3940e87b0d" (UID: "b905ef8e-61e8-4693-ae47-1b3940e87b0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:11.816841 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.816778    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-envoy" pod "b905ef8e-61e8-4693-ae47-1b3940e87b0d" (UID: "b905ef8e-61e8-4693-ae47-1b3940e87b0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.816938 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.816846    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-token" pod "b905ef8e-61e8-4693-ae47-1b3940e87b0d" (UID: "b905ef8e-61e8-4693-ae47-1b3940e87b0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.824262 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.824188    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"]
Oct 29 23:01:11.832253 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.832214    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e89c-745667d778-c8x5k"]
Oct 29 23:01:11.872246 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.872205    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"]
Oct 29 23:01:11.879957 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.879919    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-784f845595-7vhc9"]
Oct 29 23:01:11.914874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.913326    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.914874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.913356    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4pkb\" (UniqueName: \"kubernetes.io/projected/b905ef8e-61e8-4693-ae47-1b3940e87b0d-kube-api-access-r4pkb\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.914874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.913366    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.914874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.913376    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.914874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.913384    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.914874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.913392    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.914874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.913399    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b905ef8e-61e8-4693-ae47-1b3940e87b0d-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.914874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:11.913407    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b905ef8e-61e8-4693-ae47-1b3940e87b0d-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.998160 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:01:11.998059    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ae42cd_1e69_44b6_b728_fb34e15e46c7.slice/crio-conmon-81e6d2e4b33c25cb2c81b62dda6db1a217512aba84686ac0f8baa0ca4ca18acf.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-conmon-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e53e77_e3c5_4468_b830_79e768275dde.slice/crio-e9f4341b677e8dae63a2c3945bfdc17117e11cfe561456f230641c1fb253d3cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6a5323_5dab_421e_9c50_b28825aaa895.slice/crio-4b978c63699e8ab4ab4ea9f2ed09240271664b8f578990f0dc9414cf32a8e69c.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:01:12.647613 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:12.647574    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml" event={"ID":"a0fc466b-5f14-4f21-bba8-d8be8f30323f","Type":"ContainerStarted","Data":"e4ba5696da3753449dfaac8c92ad706d1ddc41ba347c7a6a76ed6012d1451326"}
Oct 29 23:01:12.648215 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:12.647677    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"
Oct 29 23:01:12.650557 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:12.650524    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6" event={"ID":"db95228d-b0d3-41df-966d-770f4f338a72","Type":"ContainerStarted","Data":"f0ce699d1f59de4a54ac032a6076203a54c787e633cbe3aa08a0be790f968b5b"}
Oct 29 23:01:12.650883 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:12.650862    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"
Oct 29 23:01:12.655485 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:12.655456    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"
Oct 29 23:01:12.655660 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:12.655441    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84" event={"ID":"b905ef8e-61e8-4693-ae47-1b3940e87b0d","Type":"ContainerDied","Data":"b5d8de0502d8bf15c01cba1aef3453df002907d8de217a67efee9a6a07f8d612"}
Oct 29 23:01:12.655839 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:12.655794    4559 scope.go:122] "RemoveContainer" containerID="4c77f80dafa1521bffc45f05ccd21021095594b8cb7e09b51fb036430c2a7822"
Oct 29 23:01:12.688588 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:12.688494    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml" podStartSLOduration=3.688471232 podStartE2EDuration="3.688471232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:12.686194894 +0000 UTC m=+2811.597638448" watchObservedRunningTime="2026-10-29 23:01:12.688471232 +0000 UTC m=+2811.599914788"
Oct 29 23:01:12.732662 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:12.732596    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6" podStartSLOduration=2.732575466 podStartE2EDuration="2.732575466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:12.725607257 +0000 UTC m=+2811.637050811" watchObservedRunningTime="2026-10-29 23:01:12.732575466 +0000 UTC m=+2811.644019020"
Oct 29 23:01:12.781084 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:12.781039    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 23:01:12.781508 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:12.781487    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 23:01:12.782193 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:12.782173    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 23:01:12.795258 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:12.795193    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"]
Oct 29 23:01:12.808582 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:12.808539    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-2kr84"]
Oct 29 23:01:13.232728 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:13.232684    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0400c09d-e610-4055-b068-c00c6f156c9c" path="/var/lib/kubelet/pods/0400c09d-e610-4055-b068-c00c6f156c9c/volumes"
Oct 29 23:01:13.233317 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:13.233293    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b93002e-a2d9-4bd4-9bb3-266576873ebb" path="/var/lib/kubelet/pods/6b93002e-a2d9-4bd4-9bb3-266576873ebb/volumes"
Oct 29 23:01:13.233874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:13.233854    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b905ef8e-61e8-4693-ae47-1b3940e87b0d" path="/var/lib/kubelet/pods/b905ef8e-61e8-4693-ae47-1b3940e87b0d/volumes"
Oct 29 23:01:13.694693 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:01:13.694656    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:01:17.211364 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:17.211324    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"]
Oct 29 23:01:21.217078 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.217035    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"]
Oct 29 23:01:21.217784 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.217769    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b905ef8e-61e8-4693-ae47-1b3940e87b0d" containerName="istio-proxy"
Oct 29 23:01:21.217871 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.217785    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b905ef8e-61e8-4693-ae47-1b3940e87b0d" containerName="istio-proxy"
Oct 29 23:01:21.217871 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.217823    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b905ef8e-61e8-4693-ae47-1b3940e87b0d" containerName="istio-proxy"
Oct 29 23:01:21.217871 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.217828    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b905ef8e-61e8-4693-ae47-1b3940e87b0d" containerName="istio-proxy"
Oct 29 23:01:21.217871 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.217848    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0400c09d-e610-4055-b068-c00c6f156c9c" containerName="payload-processing"
Oct 29 23:01:21.217871 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.217853    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0400c09d-e610-4055-b068-c00c6f156c9c" containerName="payload-processing"
Oct 29 23:01:21.217871 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.217867    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d64eb694-3958-4dec-a8cc-beaf163d64d4" containerName="revoke-keys"
Oct 29 23:01:21.217871 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.217872    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d64eb694-3958-4dec-a8cc-beaf163d64d4" containerName="revoke-keys"
Oct 29 23:01:21.218112 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.217881    4559 cpu_manager.go:414] "containerMap: removing container" podUID="de7c3068-21b6-4ae0-9f35-c60e986538c7" containerName="revoke-keys"
Oct 29 23:01:21.218112 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.217886    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7c3068-21b6-4ae0-9f35-c60e986538c7" containerName="revoke-keys"
Oct 29 23:01:21.218112 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.217911    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6b93002e-a2d9-4bd4-9bb3-266576873ebb" containerName="payload-pre-processing"
Oct 29 23:01:21.218112 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.217915    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b93002e-a2d9-4bd4-9bb3-266576873ebb" containerName="payload-pre-processing"
Oct 29 23:01:21.218112 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.218012    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b93002e-a2d9-4bd4-9bb3-266576873ebb" containerName="payload-pre-processing"
Oct 29 23:01:21.218112 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.218020    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b905ef8e-61e8-4693-ae47-1b3940e87b0d" containerName="istio-proxy"
Oct 29 23:01:21.218112 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.218033    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0400c09d-e610-4055-b068-c00c6f156c9c" containerName="payload-processing"
Oct 29 23:01:21.218112 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.218041    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d64eb694-3958-4dec-a8cc-beaf163d64d4" containerName="revoke-keys"
Oct 29 23:01:21.218112 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.218047    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7c3068-21b6-4ae0-9f35-c60e986538c7" containerName="revoke-keys"
Oct 29 23:01:21.224606 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.224574    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"
Oct 29 23:01:21.256536 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.244047    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"]
Oct 29 23:01:21.256536 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.246367    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b905ef8e-61e8-4693-ae47-1b3940e87b0d" containerName="istio-proxy"
Oct 29 23:01:21.256536 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.253157    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"]
Oct 29 23:01:21.256536 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.253273    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"
Oct 29 23:01:21.257529 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.257501    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:01:21.257644 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.257629    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:01:21.283500 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.283467    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"]
Oct 29 23:01:21.326855 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.326788    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgqhn\" (UniqueName: \"kubernetes.io/projected/989e4f47-30aa-44b7-bb2c-052ce6760a82-kube-api-access-wgqhn\") pod \"maas-api-revoke-sub-f890f139-n5qpl\" (UID: \"989e4f47-30aa-44b7-bb2c-052ce6760a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"
Oct 29 23:01:21.327038 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.326967    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989e4f47-30aa-44b7-bb2c-052ce6760a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f890f139-n5qpl\" (UID: \"989e4f47-30aa-44b7-bb2c-052ce6760a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"
Oct 29 23:01:21.327038 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.327003    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989e4f47-30aa-44b7-bb2c-052ce6760a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f890f139-n5qpl\" (UID: \"989e4f47-30aa-44b7-bb2c-052ce6760a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"
Oct 29 23:01:21.428571 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.428533    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqsw2\" (UniqueName: \"kubernetes.io/projected/f5dda999-f738-4ddc-bdc2-02a322c38723-kube-api-access-hqsw2\") pod \"maas-api-revoke-sub-f9f821fe-ll528\" (UID: \"f5dda999-f738-4ddc-bdc2-02a322c38723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"
Oct 29 23:01:21.428751 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.428658    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5dda999-f738-4ddc-bdc2-02a322c38723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9f821fe-ll528\" (UID: \"f5dda999-f738-4ddc-bdc2-02a322c38723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"
Oct 29 23:01:21.428751 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.428692    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989e4f47-30aa-44b7-bb2c-052ce6760a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f890f139-n5qpl\" (UID: \"989e4f47-30aa-44b7-bb2c-052ce6760a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"
Oct 29 23:01:21.428916 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.428777    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989e4f47-30aa-44b7-bb2c-052ce6760a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f890f139-n5qpl\" (UID: \"989e4f47-30aa-44b7-bb2c-052ce6760a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"
Oct 29 23:01:21.428996 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.428982    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5dda999-f738-4ddc-bdc2-02a322c38723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9f821fe-ll528\" (UID: \"f5dda999-f738-4ddc-bdc2-02a322c38723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"
Oct 29 23:01:21.429058 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.429046    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgqhn\" (UniqueName: \"kubernetes.io/projected/989e4f47-30aa-44b7-bb2c-052ce6760a82-kube-api-access-wgqhn\") pod \"maas-api-revoke-sub-f890f139-n5qpl\" (UID: \"989e4f47-30aa-44b7-bb2c-052ce6760a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"
Oct 29 23:01:21.429506 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.429487    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989e4f47-30aa-44b7-bb2c-052ce6760a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f890f139-n5qpl\" (UID: \"989e4f47-30aa-44b7-bb2c-052ce6760a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"
Oct 29 23:01:21.431344 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.431321    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989e4f47-30aa-44b7-bb2c-052ce6760a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f890f139-n5qpl\" (UID: \"989e4f47-30aa-44b7-bb2c-052ce6760a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"
Oct 29 23:01:21.449328 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.449290    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgqhn\" (UniqueName: \"kubernetes.io/projected/989e4f47-30aa-44b7-bb2c-052ce6760a82-kube-api-access-wgqhn\") pod \"maas-api-revoke-sub-f890f139-n5qpl\" (UID: \"989e4f47-30aa-44b7-bb2c-052ce6760a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"
Oct 29 23:01:21.531440 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.531336    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5dda999-f738-4ddc-bdc2-02a322c38723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9f821fe-ll528\" (UID: \"f5dda999-f738-4ddc-bdc2-02a322c38723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"
Oct 29 23:01:21.531620 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.531449    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqsw2\" (UniqueName: \"kubernetes.io/projected/f5dda999-f738-4ddc-bdc2-02a322c38723-kube-api-access-hqsw2\") pod \"maas-api-revoke-sub-f9f821fe-ll528\" (UID: \"f5dda999-f738-4ddc-bdc2-02a322c38723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"
Oct 29 23:01:21.531620 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.531494    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5dda999-f738-4ddc-bdc2-02a322c38723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9f821fe-ll528\" (UID: \"f5dda999-f738-4ddc-bdc2-02a322c38723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"
Oct 29 23:01:21.532148 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.532121    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5dda999-f738-4ddc-bdc2-02a322c38723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9f821fe-ll528\" (UID: \"f5dda999-f738-4ddc-bdc2-02a322c38723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"
Oct 29 23:01:21.534274 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.534248    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5dda999-f738-4ddc-bdc2-02a322c38723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9f821fe-ll528\" (UID: \"f5dda999-f738-4ddc-bdc2-02a322c38723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"
Oct 29 23:01:21.544938 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.544896    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"
Oct 29 23:01:21.556494 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.556456    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqsw2\" (UniqueName: \"kubernetes.io/projected/f5dda999-f738-4ddc-bdc2-02a322c38723-kube-api-access-hqsw2\") pod \"maas-api-revoke-sub-f9f821fe-ll528\" (UID: \"f5dda999-f738-4ddc-bdc2-02a322c38723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"
Oct 29 23:01:21.593090 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.593050    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"
Oct 29 23:01:21.895863 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:01:21.895789    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989e4f47_30aa_44b7_bb2c_052ce6760a82.slice/crio-b464bf48d4a16b7f8b39460bbe228631350097af1ea2f8b504e7685bc6826771 WatchSource:0}: Error finding container b464bf48d4a16b7f8b39460bbe228631350097af1ea2f8b504e7685bc6826771: Status 404 returned error can't find the container with id b464bf48d4a16b7f8b39460bbe228631350097af1ea2f8b504e7685bc6826771
Oct 29 23:01:21.898725 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.898697    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"]
Oct 29 23:01:21.924165 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:21.924134    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"]
Oct 29 23:01:21.931043 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:01:21.931015    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5dda999_f738_4ddc_bdc2_02a322c38723.slice/crio-cbbf650c88a829f5b062d811a0441ee9fa89580d4ca0085ff0f2c091dc307636 WatchSource:0}: Error finding container cbbf650c88a829f5b062d811a0441ee9fa89580d4ca0085ff0f2c091dc307636: Status 404 returned error can't find the container with id cbbf650c88a829f5b062d811a0441ee9fa89580d4ca0085ff0f2c091dc307636
Oct 29 23:01:22.212249 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:22.212205    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj" podUID="ecd94d51-9423-41c3-8ab2-e94b52ab5904" containerName="payload-pre-processing" containerID="cri-o://536623d37bcad3265f823d0d48ec46bb5096d871c44b055c3811335a8e0ac5fa" gracePeriod=25
Oct 29 23:01:22.778949 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:22.778914    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528" event={"ID":"f5dda999-f738-4ddc-bdc2-02a322c38723","Type":"ContainerStarted","Data":"31f20ed073933aa4cdbf19be9e3115a48d2dfa505b5b447ab7a8cb131c071542"}
Oct 29 23:01:22.779478 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:22.778988    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528" event={"ID":"f5dda999-f738-4ddc-bdc2-02a322c38723","Type":"ContainerStarted","Data":"cbbf650c88a829f5b062d811a0441ee9fa89580d4ca0085ff0f2c091dc307636"}
Oct 29 23:01:22.780756 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:22.780716    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl" event={"ID":"989e4f47-30aa-44b7-bb2c-052ce6760a82","Type":"ContainerStarted","Data":"8cd2f4cae8de9694673bb093ebaac436a0ebe63bff24d66fd74c4aeecba94353"}
Oct 29 23:01:22.780756 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:22.780759    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl" event={"ID":"989e4f47-30aa-44b7-bb2c-052ce6760a82","Type":"ContainerStarted","Data":"b464bf48d4a16b7f8b39460bbe228631350097af1ea2f8b504e7685bc6826771"}
Oct 29 23:01:22.821060 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:22.821004    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528" podStartSLOduration=1.8209884299999999 podStartE2EDuration="1.82098843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:22.817387127 +0000 UTC m=+2821.728830682" watchObservedRunningTime="2026-10-29 23:01:22.82098843 +0000 UTC m=+2821.732431983"
Oct 29 23:01:22.862242 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:22.862169    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl" podStartSLOduration=1.862151307 podStartE2EDuration="1.862151307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:22.85969871 +0000 UTC m=+2821.771142289" watchObservedRunningTime="2026-10-29 23:01:22.862151307 +0000 UTC m=+2821.773594860"
Oct 29 23:01:23.668555 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:23.668503    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"
Oct 29 23:01:23.669961 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:23.669936    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"
Oct 29 23:01:23.804966 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:23.804924    4559 generic.go:356] "Generic (PLEG): container finished" podID="f5dda999-f738-4ddc-bdc2-02a322c38723" containerID="31f20ed073933aa4cdbf19be9e3115a48d2dfa505b5b447ab7a8cb131c071542" exitCode=0
Oct 29 23:01:23.805420 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:23.805105    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528" event={"ID":"f5dda999-f738-4ddc-bdc2-02a322c38723","Type":"ContainerDied","Data":"31f20ed073933aa4cdbf19be9e3115a48d2dfa505b5b447ab7a8cb131c071542"}
Oct 29 23:01:23.808484 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:23.808195    4559 generic.go:356] "Generic (PLEG): container finished" podID="989e4f47-30aa-44b7-bb2c-052ce6760a82" containerID="8cd2f4cae8de9694673bb093ebaac436a0ebe63bff24d66fd74c4aeecba94353" exitCode=0
Oct 29 23:01:23.808484 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:23.808240    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl" event={"ID":"989e4f47-30aa-44b7-bb2c-052ce6760a82","Type":"ContainerDied","Data":"8cd2f4cae8de9694673bb093ebaac436a0ebe63bff24d66fd74c4aeecba94353"}
Oct 29 23:01:23.816830 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:23.816759    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"]
Oct 29 23:01:23.817202 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:23.817116    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p" podUID="ce60109b-2f7b-462b-ac57-9691a2e6b6fc" containerName="payload-processing" containerID="cri-o://cdfdbab14ace0abe8e1fe05c3a925262dd456c66c64045b1556718680bdccba2" gracePeriod=30
Oct 29 23:01:23.830377 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:23.830345    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"]
Oct 29 23:01:23.830667 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:23.830641    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5" podUID="ab0aace4-e648-47e5-b31e-2203a97f5464" containerName="payload-pre-processing" containerID="cri-o://f8866a8baef8247ae23212cae6b1f8ab3e6b70ccd914d484d6b7fcecd82a8fb7" gracePeriod=30
Oct 29 23:01:23.912732 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:23.912690    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"]
Oct 29 23:01:23.917852 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:23.917792    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-83f120-6r6xp"]
Oct 29 23:01:24.188887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.188854    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"
Oct 29 23:01:24.190829 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.190762    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce60109b-2f7b-462b-ac57-9691a2e6b6fc-kube-api-access-lkvlz\" (UniqueName: \"kubernetes.io/projected/ce60109b-2f7b-462b-ac57-9691a2e6b6fc-kube-api-access-lkvlz\") pod \"ce60109b-2f7b-462b-ac57-9691a2e6b6fc\" (UID: \"ce60109b-2f7b-462b-ac57-9691a2e6b6fc\") "
Oct 29 23:01:24.191366 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.191347    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce60109b-2f7b-462b-ac57-9691a2e6b6fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce60109b-2f7b-462b-ac57-9691a2e6b6fc-plugins-config-volume\") pod \"ce60109b-2f7b-462b-ac57-9691a2e6b6fc\" (UID: \"ce60109b-2f7b-462b-ac57-9691a2e6b6fc\") "
Oct 29 23:01:24.191779 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.191748    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce60109b-2f7b-462b-ac57-9691a2e6b6fc-plugins-config-volume" pod "ce60109b-2f7b-462b-ac57-9691a2e6b6fc" (UID: "ce60109b-2f7b-462b-ac57-9691a2e6b6fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:24.196874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.193314    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce60109b-2f7b-462b-ac57-9691a2e6b6fc-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.196874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.194835    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce60109b-2f7b-462b-ac57-9691a2e6b6fc-kube-api-access-lkvlz" pod "ce60109b-2f7b-462b-ac57-9691a2e6b6fc" (UID: "ce60109b-2f7b-462b-ac57-9691a2e6b6fc"). InnerVolumeSpecName "kube-api-access-lkvlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:24.273340 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.273285    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"
Oct 29 23:01:24.289159 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.289124    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"]
Oct 29 23:01:24.289403 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.289373    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d" podUID="00aa21ab-c4a0-460c-9647-ba3fece700cc" containerName="payload-pre-processing" containerID="cri-o://3933d302c9f33235ca7fa5b68cab508498d15014e84cb6952a796c14ffcbcbbf" gracePeriod=30
Oct 29 23:01:24.308541 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.307924    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0aace4-e648-47e5-b31e-2203a97f5464-kube-api-access-mfhr5\" (UniqueName: \"kubernetes.io/projected/ab0aace4-e648-47e5-b31e-2203a97f5464-kube-api-access-mfhr5\") pod \"ab0aace4-e648-47e5-b31e-2203a97f5464\" (UID: \"ab0aace4-e648-47e5-b31e-2203a97f5464\") "
Oct 29 23:01:24.308541 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.308076    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab0aace4-e648-47e5-b31e-2203a97f5464-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab0aace4-e648-47e5-b31e-2203a97f5464-plugins-config-volume\") pod \"ab0aace4-e648-47e5-b31e-2203a97f5464\" (UID: \"ab0aace4-e648-47e5-b31e-2203a97f5464\") "
Oct 29 23:01:24.309262 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.309095    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0aace4-e648-47e5-b31e-2203a97f5464-plugins-config-volume" pod "ab0aace4-e648-47e5-b31e-2203a97f5464" (UID: "ab0aace4-e648-47e5-b31e-2203a97f5464"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:24.311553 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.311339    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab0aace4-e648-47e5-b31e-2203a97f5464-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.311553 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.311399    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkvlz\" (UniqueName: \"kubernetes.io/projected/ce60109b-2f7b-462b-ac57-9691a2e6b6fc-kube-api-access-lkvlz\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.342974 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.342941    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0aace4-e648-47e5-b31e-2203a97f5464-kube-api-access-mfhr5" pod "ab0aace4-e648-47e5-b31e-2203a97f5464" (UID: "ab0aace4-e648-47e5-b31e-2203a97f5464"). InnerVolumeSpecName "kube-api-access-mfhr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:24.413379 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.413348    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfhr5\" (UniqueName: \"kubernetes.io/projected/ab0aace4-e648-47e5-b31e-2203a97f5464-kube-api-access-mfhr5\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.443641 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.443600    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"]
Oct 29 23:01:24.450732 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.450698    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b8f54c-s2v9x"]
Oct 29 23:01:24.858656 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.858621    4559 generic.go:356] "Generic (PLEG): container finished" podID="ab0aace4-e648-47e5-b31e-2203a97f5464" containerID="f8866a8baef8247ae23212cae6b1f8ab3e6b70ccd914d484d6b7fcecd82a8fb7" exitCode=0
Oct 29 23:01:24.859323 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.858775    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5" event={"ID":"ab0aace4-e648-47e5-b31e-2203a97f5464","Type":"ContainerDied","Data":"f8866a8baef8247ae23212cae6b1f8ab3e6b70ccd914d484d6b7fcecd82a8fb7"}
Oct 29 23:01:24.859533 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.859502    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5" event={"ID":"ab0aace4-e648-47e5-b31e-2203a97f5464","Type":"ContainerDied","Data":"ba999a42bdd697ef69723966270c011cebb75a18d2cf116aab03b404ccb323c0"}
Oct 29 23:01:24.859703 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.859684    4559 scope.go:122] "RemoveContainer" containerID="f8866a8baef8247ae23212cae6b1f8ab3e6b70ccd914d484d6b7fcecd82a8fb7"
Oct 29 23:01:24.859944 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.858923    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"
Oct 29 23:01:24.891273 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.891232    4559 generic.go:356] "Generic (PLEG): container finished" podID="00aa21ab-c4a0-460c-9647-ba3fece700cc" containerID="3933d302c9f33235ca7fa5b68cab508498d15014e84cb6952a796c14ffcbcbbf" exitCode=0
Oct 29 23:01:24.891503 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.891484    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d" event={"ID":"00aa21ab-c4a0-460c-9647-ba3fece700cc","Type":"ContainerDied","Data":"3933d302c9f33235ca7fa5b68cab508498d15014e84cb6952a796c14ffcbcbbf"}
Oct 29 23:01:24.891608 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.891567    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d" event={"ID":"00aa21ab-c4a0-460c-9647-ba3fece700cc","Type":"ContainerDied","Data":"dab1505cca6d9cc8c6f462071b323dd13ce9623787af0e033ad696a93cd0263d"}
Oct 29 23:01:24.891608 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.891586    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dab1505cca6d9cc8c6f462071b323dd13ce9623787af0e033ad696a93cd0263d"
Oct 29 23:01:24.893473 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.893437    4559 generic.go:356] "Generic (PLEG): container finished" podID="ce60109b-2f7b-462b-ac57-9691a2e6b6fc" containerID="cdfdbab14ace0abe8e1fe05c3a925262dd456c66c64045b1556718680bdccba2" exitCode=0
Oct 29 23:01:24.893636 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.893501    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"
Oct 29 23:01:24.893636 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.893515    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p" event={"ID":"ce60109b-2f7b-462b-ac57-9691a2e6b6fc","Type":"ContainerDied","Data":"cdfdbab14ace0abe8e1fe05c3a925262dd456c66c64045b1556718680bdccba2"}
Oct 29 23:01:24.893636 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.893599    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p" event={"ID":"ce60109b-2f7b-462b-ac57-9691a2e6b6fc","Type":"ContainerDied","Data":"30a732f8a2a891d0ddab355a2ef4beedc3fb22ee8d6322442296aa949464923a"}
Oct 29 23:01:24.894121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.894097    4559 scope.go:122] "RemoveContainer" containerID="f8866a8baef8247ae23212cae6b1f8ab3e6b70ccd914d484d6b7fcecd82a8fb7"
Oct 29 23:01:24.895421 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:01:24.895376    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8866a8baef8247ae23212cae6b1f8ab3e6b70ccd914d484d6b7fcecd82a8fb7\": container with ID starting with f8866a8baef8247ae23212cae6b1f8ab3e6b70ccd914d484d6b7fcecd82a8fb7 not found: ID does not exist" containerID="f8866a8baef8247ae23212cae6b1f8ab3e6b70ccd914d484d6b7fcecd82a8fb7"
Oct 29 23:01:24.895629 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.895442    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8866a8baef8247ae23212cae6b1f8ab3e6b70ccd914d484d6b7fcecd82a8fb7"} err="failed to get container status \"f8866a8baef8247ae23212cae6b1f8ab3e6b70ccd914d484d6b7fcecd82a8fb7\": rpc error: code = NotFound desc = could not find container \"f8866a8baef8247ae23212cae6b1f8ab3e6b70ccd914d484d6b7fcecd82a8fb7\": container with ID starting with f8866a8baef8247ae23212cae6b1f8ab3e6b70ccd914d484d6b7fcecd82a8fb7 not found: ID does not exist"
Oct 29 23:01:24.895629 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.895468    4559 scope.go:122] "RemoveContainer" containerID="cdfdbab14ace0abe8e1fe05c3a925262dd456c66c64045b1556718680bdccba2"
Oct 29 23:01:24.896437 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.896410    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"
Oct 29 23:01:24.910914 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.910841    4559 scope.go:122] "RemoveContainer" containerID="cdfdbab14ace0abe8e1fe05c3a925262dd456c66c64045b1556718680bdccba2"
Oct 29 23:01:24.912307 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:01:24.912278    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdfdbab14ace0abe8e1fe05c3a925262dd456c66c64045b1556718680bdccba2\": container with ID starting with cdfdbab14ace0abe8e1fe05c3a925262dd456c66c64045b1556718680bdccba2 not found: ID does not exist" containerID="cdfdbab14ace0abe8e1fe05c3a925262dd456c66c64045b1556718680bdccba2"
Oct 29 23:01:24.912463 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.912315    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdfdbab14ace0abe8e1fe05c3a925262dd456c66c64045b1556718680bdccba2"} err="failed to get container status \"cdfdbab14ace0abe8e1fe05c3a925262dd456c66c64045b1556718680bdccba2\": rpc error: code = NotFound desc = could not find container \"cdfdbab14ace0abe8e1fe05c3a925262dd456c66c64045b1556718680bdccba2\": container with ID starting with cdfdbab14ace0abe8e1fe05c3a925262dd456c66c64045b1556718680bdccba2 not found: ID does not exist"
Oct 29 23:01:24.954344 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.951965    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"]
Oct 29 23:01:24.958171 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:24.958140    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-7c969c8947-9wpb5"]
Oct 29 23:01:25.028030 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.027995    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"]
Oct 29 23:01:25.048692 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.038407    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00aa21ab-c4a0-460c-9647-ba3fece700cc-kube-api-access-6h5b2\" (UniqueName: \"kubernetes.io/projected/00aa21ab-c4a0-460c-9647-ba3fece700cc-kube-api-access-6h5b2\") pod \"00aa21ab-c4a0-460c-9647-ba3fece700cc\" (UID: \"00aa21ab-c4a0-460c-9647-ba3fece700cc\") "
Oct 29 23:01:25.048692 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.038588    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00aa21ab-c4a0-460c-9647-ba3fece700cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00aa21ab-c4a0-460c-9647-ba3fece700cc-plugins-config-volume\") pod \"00aa21ab-c4a0-460c-9647-ba3fece700cc\" (UID: \"00aa21ab-c4a0-460c-9647-ba3fece700cc\") "
Oct 29 23:01:25.048692 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.039675    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00aa21ab-c4a0-460c-9647-ba3fece700cc-plugins-config-volume" pod "00aa21ab-c4a0-460c-9647-ba3fece700cc" (UID: "00aa21ab-c4a0-460c-9647-ba3fece700cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:25.048692 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.042594    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b8f54c-6dd8f8fcbb-8v49p"]
Oct 29 23:01:25.084507 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.084466    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00aa21ab-c4a0-460c-9647-ba3fece700cc-kube-api-access-6h5b2" pod "00aa21ab-c4a0-460c-9647-ba3fece700cc" (UID: "00aa21ab-c4a0-460c-9647-ba3fece700cc"). InnerVolumeSpecName "kube-api-access-6h5b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:25.119968 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.119922    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"
Oct 29 23:01:25.140981 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.140514    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h5b2\" (UniqueName: \"kubernetes.io/projected/00aa21ab-c4a0-460c-9647-ba3fece700cc-kube-api-access-6h5b2\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.140981 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.140545    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00aa21ab-c4a0-460c-9647-ba3fece700cc-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.140981 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.140756    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"
Oct 29 23:01:25.230792 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.230760    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ae1a35-1d04-478b-be37-75c897b761c0" path="/var/lib/kubelet/pods/11ae1a35-1d04-478b-be37-75c897b761c0/volumes"
Oct 29 23:01:25.231399 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.231381    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0aace4-e648-47e5-b31e-2203a97f5464" path="/var/lib/kubelet/pods/ab0aace4-e648-47e5-b31e-2203a97f5464/volumes"
Oct 29 23:01:25.231901 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.231883    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce60109b-2f7b-462b-ac57-9691a2e6b6fc" path="/var/lib/kubelet/pods/ce60109b-2f7b-462b-ac57-9691a2e6b6fc/volumes"
Oct 29 23:01:25.232644 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.232627    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d64eb694-3958-4dec-a8cc-beaf163d64d4" path="/var/lib/kubelet/pods/d64eb694-3958-4dec-a8cc-beaf163d64d4/volumes"
Oct 29 23:01:25.241262 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.241213    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989e4f47-30aa-44b7-bb2c-052ce6760a82-kube-api-access-wgqhn\" (UniqueName: \"kubernetes.io/projected/989e4f47-30aa-44b7-bb2c-052ce6760a82-kube-api-access-wgqhn\") pod \"989e4f47-30aa-44b7-bb2c-052ce6760a82\" (UID: \"989e4f47-30aa-44b7-bb2c-052ce6760a82\") "
Oct 29 23:01:25.241422 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.241361    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/989e4f47-30aa-44b7-bb2c-052ce6760a82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989e4f47-30aa-44b7-bb2c-052ce6760a82-maas-api-xks-ca\") pod \"989e4f47-30aa-44b7-bb2c-052ce6760a82\" (UID: \"989e4f47-30aa-44b7-bb2c-052ce6760a82\") "
Oct 29 23:01:25.241422 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.241414    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5dda999-f738-4ddc-bdc2-02a322c38723-kube-api-access-hqsw2\" (UniqueName: \"kubernetes.io/projected/f5dda999-f738-4ddc-bdc2-02a322c38723-kube-api-access-hqsw2\") pod \"f5dda999-f738-4ddc-bdc2-02a322c38723\" (UID: \"f5dda999-f738-4ddc-bdc2-02a322c38723\") "
Oct 29 23:01:25.241569 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.241449    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5dda999-f738-4ddc-bdc2-02a322c38723-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5dda999-f738-4ddc-bdc2-02a322c38723-maas-api-xks-ca\") pod \"f5dda999-f738-4ddc-bdc2-02a322c38723\" (UID: \"f5dda999-f738-4ddc-bdc2-02a322c38723\") "
Oct 29 23:01:25.241569 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.241545    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/989e4f47-30aa-44b7-bb2c-052ce6760a82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989e4f47-30aa-44b7-bb2c-052ce6760a82-maas-api-openshift-ca\") pod \"989e4f47-30aa-44b7-bb2c-052ce6760a82\" (UID: \"989e4f47-30aa-44b7-bb2c-052ce6760a82\") "
Oct 29 23:01:25.241686 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.241589    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5dda999-f738-4ddc-bdc2-02a322c38723-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5dda999-f738-4ddc-bdc2-02a322c38723-maas-api-openshift-ca\") pod \"f5dda999-f738-4ddc-bdc2-02a322c38723\" (UID: \"f5dda999-f738-4ddc-bdc2-02a322c38723\") "
Oct 29 23:01:25.244474 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.242085    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989e4f47-30aa-44b7-bb2c-052ce6760a82-maas-api-openshift-ca" pod "989e4f47-30aa-44b7-bb2c-052ce6760a82" (UID: "989e4f47-30aa-44b7-bb2c-052ce6760a82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:25.244474 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.242349    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5dda999-f738-4ddc-bdc2-02a322c38723-maas-api-openshift-ca" pod "f5dda999-f738-4ddc-bdc2-02a322c38723" (UID: "f5dda999-f738-4ddc-bdc2-02a322c38723"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:25.244474 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.242633    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989e4f47-30aa-44b7-bb2c-052ce6760a82-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.244474 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.242657    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5dda999-f738-4ddc-bdc2-02a322c38723-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.245108 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.244985    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989e4f47-30aa-44b7-bb2c-052ce6760a82-kube-api-access-wgqhn" pod "989e4f47-30aa-44b7-bb2c-052ce6760a82" (UID: "989e4f47-30aa-44b7-bb2c-052ce6760a82"). InnerVolumeSpecName "kube-api-access-wgqhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:25.245609 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.245564    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/989e4f47-30aa-44b7-bb2c-052ce6760a82-maas-api-xks-ca" pod "989e4f47-30aa-44b7-bb2c-052ce6760a82" (UID: "989e4f47-30aa-44b7-bb2c-052ce6760a82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:25.245710 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.245677    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5dda999-f738-4ddc-bdc2-02a322c38723-maas-api-xks-ca" pod "f5dda999-f738-4ddc-bdc2-02a322c38723" (UID: "f5dda999-f738-4ddc-bdc2-02a322c38723"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:25.247001 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.246956    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5dda999-f738-4ddc-bdc2-02a322c38723-kube-api-access-hqsw2" pod "f5dda999-f738-4ddc-bdc2-02a322c38723" (UID: "f5dda999-f738-4ddc-bdc2-02a322c38723"). InnerVolumeSpecName "kube-api-access-hqsw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:25.257596 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.257556    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"]
Oct 29 23:01:25.262419 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.262387    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7be71d-92sdg"]
Oct 29 23:01:25.345073 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.344964    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgqhn\" (UniqueName: \"kubernetes.io/projected/989e4f47-30aa-44b7-bb2c-052ce6760a82-kube-api-access-wgqhn\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.345073 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.345001    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989e4f47-30aa-44b7-bb2c-052ce6760a82-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.345073 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.345016    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqsw2\" (UniqueName: \"kubernetes.io/projected/f5dda999-f738-4ddc-bdc2-02a322c38723-kube-api-access-hqsw2\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.345073 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.345030    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5dda999-f738-4ddc-bdc2-02a322c38723-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.905451 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.905414    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528" event={"ID":"f5dda999-f738-4ddc-bdc2-02a322c38723","Type":"ContainerDied","Data":"cbbf650c88a829f5b062d811a0441ee9fa89580d4ca0085ff0f2c091dc307636"}
Oct 29 23:01:25.905451 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.905442    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"
Oct 29 23:01:25.905977 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.905479    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbbf650c88a829f5b062d811a0441ee9fa89580d4ca0085ff0f2c091dc307636"
Oct 29 23:01:25.908001 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.907977    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"
Oct 29 23:01:25.908001 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.907983    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl" event={"ID":"989e4f47-30aa-44b7-bb2c-052ce6760a82","Type":"ContainerDied","Data":"b464bf48d4a16b7f8b39460bbe228631350097af1ea2f8b504e7685bc6826771"}
Oct 29 23:01:25.908206 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.908023    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b464bf48d4a16b7f8b39460bbe228631350097af1ea2f8b504e7685bc6826771"
Oct 29 23:01:25.909432 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.909409    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"
Oct 29 23:01:25.979020 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.978984    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"]
Oct 29 23:01:25.990755 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:25.990723    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7be71d-865cd9bc7f-52m9d"]
Oct 29 23:01:26.576167 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:26.576118    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main" containerID="cri-o://c14e601cf13401fd733920ccd520cd01cca892f63c229a14cb50b21b4cad9afb" gracePeriod=45
Oct 29 23:01:26.632841 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:26.632775    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main" containerID="cri-o://91b6aac3f7c570c7478709b1d31627e07a983b81a8d2f500ab49ba84da2c7a1b" gracePeriod=45
Oct 29 23:01:26.927324 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:26.926976    4559 generic.go:356] "Generic (PLEG): container finished" podID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerID="91b6aac3f7c570c7478709b1d31627e07a983b81a8d2f500ab49ba84da2c7a1b" exitCode=0
Oct 29 23:01:26.927324 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:26.927058    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" event={"ID":"daabb28c-a3d4-43f5-85d7-b09141c8a4fd","Type":"ContainerDied","Data":"91b6aac3f7c570c7478709b1d31627e07a983b81a8d2f500ab49ba84da2c7a1b"}
Oct 29 23:01:26.933697 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:26.933513    4559 generic.go:356] "Generic (PLEG): container finished" podID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerID="c14e601cf13401fd733920ccd520cd01cca892f63c229a14cb50b21b4cad9afb" exitCode=0
Oct 29 23:01:26.933895 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:26.933865    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" event={"ID":"779ec9d1-52de-4f60-9f5e-4be62ceb311f","Type":"ContainerDied","Data":"c14e601cf13401fd733920ccd520cd01cca892f63c229a14cb50b21b4cad9afb"}
Oct 29 23:01:26.933961 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:26.933930    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224" event={"ID":"779ec9d1-52de-4f60-9f5e-4be62ceb311f","Type":"ContainerDied","Data":"ac630464464f942234be329f4fd675f9c33ad7ed0202c400207601706756ecdc"}
Oct 29 23:01:26.933961 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:26.933948    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac630464464f942234be329f4fd675f9c33ad7ed0202c400207601706756ecdc"
Oct 29 23:01:26.935110 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:26.935080    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 23:01:26.995017 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:26.994977    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 23:01:27.035894 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.035857    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-dshm\") pod \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") "
Oct 29 23:01:27.036080 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.035912    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-kube-api-access-7xx2n\" (UniqueName: \"kubernetes.io/projected/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-kube-api-access-7xx2n\") pod \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") "
Oct 29 23:01:27.036080 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.036023    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-model-cache\") pod \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") "
Oct 29 23:01:27.036221 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.036081    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-tmp-dir\") pod \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") "
Oct 29 23:01:27.036221 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.036148    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-tls-certs\") pod \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") "
Oct 29 23:01:27.036545 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.036327    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/779ec9d1-52de-4f60-9f5e-4be62ceb311f-kube-api-access-sznh9\" (UniqueName: \"kubernetes.io/projected/779ec9d1-52de-4f60-9f5e-4be62ceb311f-kube-api-access-sznh9\") pod \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") "
Oct 29 23:01:27.036545 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.036362    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-tmp-dir\") pod \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") "
Oct 29 23:01:27.036545 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.036427    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-home\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-home\") pod \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") "
Oct 29 23:01:27.036545 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.036464    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-model-cache\") pod \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\" (UID: \"daabb28c-a3d4-43f5-85d7-b09141c8a4fd\") "
Oct 29 23:01:27.036545 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.036490    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-home\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-home\") pod \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") "
Oct 29 23:01:27.036545 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.036515    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-dshm\") pod \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") "
Oct 29 23:01:27.036545 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.036546    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/779ec9d1-52de-4f60-9f5e-4be62ceb311f-tls-certs\" (UniqueName: \"kubernetes.io/secret/779ec9d1-52de-4f60-9f5e-4be62ceb311f-tls-certs\") pod \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\" (UID: \"779ec9d1-52de-4f60-9f5e-4be62ceb311f\") "
Oct 29 23:01:27.037537 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.037050    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-model-cache" pod "779ec9d1-52de-4f60-9f5e-4be62ceb311f" (UID: "779ec9d1-52de-4f60-9f5e-4be62ceb311f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:27.037537 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.037244    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:27.039402 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.038675    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-model-cache" pod "daabb28c-a3d4-43f5-85d7-b09141c8a4fd" (UID: "daabb28c-a3d4-43f5-85d7-b09141c8a4fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:27.039402 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.038940    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-home" pod "daabb28c-a3d4-43f5-85d7-b09141c8a4fd" (UID: "daabb28c-a3d4-43f5-85d7-b09141c8a4fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:27.039402 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.039122    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-home" pod "779ec9d1-52de-4f60-9f5e-4be62ceb311f" (UID: "779ec9d1-52de-4f60-9f5e-4be62ceb311f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:27.039912 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.039863    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-tmp-dir" pod "779ec9d1-52de-4f60-9f5e-4be62ceb311f" (UID: "779ec9d1-52de-4f60-9f5e-4be62ceb311f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:27.040202 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.040172    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-tmp-dir" pod "daabb28c-a3d4-43f5-85d7-b09141c8a4fd" (UID: "daabb28c-a3d4-43f5-85d7-b09141c8a4fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:27.060066 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.050213    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779ec9d1-52de-4f60-9f5e-4be62ceb311f-kube-api-access-sznh9" pod "779ec9d1-52de-4f60-9f5e-4be62ceb311f" (UID: "779ec9d1-52de-4f60-9f5e-4be62ceb311f"). InnerVolumeSpecName "kube-api-access-sznh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:27.060066 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.050337    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-tls-certs" pod "daabb28c-a3d4-43f5-85d7-b09141c8a4fd" (UID: "daabb28c-a3d4-43f5-85d7-b09141c8a4fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:27.060066 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.050427    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-dshm" pod "daabb28c-a3d4-43f5-85d7-b09141c8a4fd" (UID: "daabb28c-a3d4-43f5-85d7-b09141c8a4fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:27.060066 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.051838    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779ec9d1-52de-4f60-9f5e-4be62ceb311f-tls-certs" pod "779ec9d1-52de-4f60-9f5e-4be62ceb311f" (UID: "779ec9d1-52de-4f60-9f5e-4be62ceb311f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:27.061274 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.061189    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-dshm" pod "779ec9d1-52de-4f60-9f5e-4be62ceb311f" (UID: "779ec9d1-52de-4f60-9f5e-4be62ceb311f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:27.061274 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.061241    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-kube-api-access-7xx2n" pod "daabb28c-a3d4-43f5-85d7-b09141c8a4fd" (UID: "daabb28c-a3d4-43f5-85d7-b09141c8a4fd"). InnerVolumeSpecName "kube-api-access-7xx2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:27.139228 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.139134    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:27.139228 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.139168    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:27.139228 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.139179    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779ec9d1-52de-4f60-9f5e-4be62ceb311f-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:27.139228 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.139193    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:27.139228 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.139206    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xx2n\" (UniqueName: \"kubernetes.io/projected/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-kube-api-access-7xx2n\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:27.139228 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.139218    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779ec9d1-52de-4f60-9f5e-4be62ceb311f-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:27.139228 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.139232    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:27.139601 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.139243    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sznh9\" (UniqueName: \"kubernetes.io/projected/779ec9d1-52de-4f60-9f5e-4be62ceb311f-kube-api-access-sznh9\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:27.139601 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.139255    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:27.139601 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.139268    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:27.139601 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.139279    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daabb28c-a3d4-43f5-85d7-b09141c8a4fd-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:27.233682 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.233472    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00aa21ab-c4a0-460c-9647-ba3fece700cc" path="/var/lib/kubelet/pods/00aa21ab-c4a0-460c-9647-ba3fece700cc/volumes"
Oct 29 23:01:27.234373 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.234233    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="199dffea-70ca-4137-90f7-9349eeac1660" path="/var/lib/kubelet/pods/199dffea-70ca-4137-90f7-9349eeac1660/volumes"
Oct 29 23:01:27.364836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.350468    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"]
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372451    4559 cpu_manager.go:414] "containerMap: removing container" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372480    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372500    4559 cpu_manager.go:414] "containerMap: removing container" podUID="00aa21ab-c4a0-460c-9647-ba3fece700cc" containerName="payload-pre-processing"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372520    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00aa21ab-c4a0-460c-9647-ba3fece700cc" containerName="payload-pre-processing"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372543    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ce60109b-2f7b-462b-ac57-9691a2e6b6fc" containerName="payload-processing"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372550    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce60109b-2f7b-462b-ac57-9691a2e6b6fc" containerName="payload-processing"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372566    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ab0aace4-e648-47e5-b31e-2203a97f5464" containerName="payload-pre-processing"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372574    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0aace4-e648-47e5-b31e-2203a97f5464" containerName="payload-pre-processing"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372605    4559 cpu_manager.go:414] "containerMap: removing container" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372613    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372678    4559 cpu_manager.go:414] "containerMap: removing container" podUID="989e4f47-30aa-44b7-bb2c-052ce6760a82" containerName="revoke-keys"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372685    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989e4f47-30aa-44b7-bb2c-052ce6760a82" containerName="revoke-keys"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372700    4559 cpu_manager.go:414] "containerMap: removing container" podUID="f5dda999-f738-4ddc-bdc2-02a322c38723" containerName="revoke-keys"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372709    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5dda999-f738-4ddc-bdc2-02a322c38723" containerName="revoke-keys"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372930    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" containerName="main"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372943    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="00aa21ab-c4a0-460c-9647-ba3fece700cc" containerName="payload-pre-processing"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372957    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce60109b-2f7b-462b-ac57-9691a2e6b6fc" containerName="payload-processing"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372965    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0aace4-e648-47e5-b31e-2203a97f5464" containerName="payload-pre-processing"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.372988    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" containerName="main"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.373002    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="989e4f47-30aa-44b7-bb2c-052ce6760a82" containerName="revoke-keys"
Oct 29 23:01:27.385887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.373012    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5dda999-f738-4ddc-bdc2-02a322c38723" containerName="revoke-keys"
Oct 29 23:01:27.392914 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.392884    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"]
Oct 29 23:01:27.393296 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.393214    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"
Oct 29 23:01:27.419908 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.417977    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-75b68d\""
Oct 29 23:01:27.438182 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.435932    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-75b68d-dockercfg-lpp4d\""
Oct 29 23:01:27.484575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.484517    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkbqp\" (UniqueName: \"kubernetes.io/projected/732da306-43ba-4e56-a01f-70532d0211cb-kube-api-access-kkbqp\") pod \"payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7\" (UID: \"732da306-43ba-4e56-a01f-70532d0211cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"
Oct 29 23:01:27.484822 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.484658    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732da306-43ba-4e56-a01f-70532d0211cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7\" (UID: \"732da306-43ba-4e56-a01f-70532d0211cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"
Oct 29 23:01:27.586503 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.586460    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkbqp\" (UniqueName: \"kubernetes.io/projected/732da306-43ba-4e56-a01f-70532d0211cb-kube-api-access-kkbqp\") pod \"payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7\" (UID: \"732da306-43ba-4e56-a01f-70532d0211cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"
Oct 29 23:01:27.586678 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.586563    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732da306-43ba-4e56-a01f-70532d0211cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7\" (UID: \"732da306-43ba-4e56-a01f-70532d0211cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"
Oct 29 23:01:27.587310 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.587287    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732da306-43ba-4e56-a01f-70532d0211cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7\" (UID: \"732da306-43ba-4e56-a01f-70532d0211cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"
Oct 29 23:01:27.614302 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.614266    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkbqp\" (UniqueName: \"kubernetes.io/projected/732da306-43ba-4e56-a01f-70532d0211cb-kube-api-access-kkbqp\") pod \"payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7\" (UID: \"732da306-43ba-4e56-a01f-70532d0211cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"
Oct 29 23:01:27.757329 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.751417    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"
Oct 29 23:01:27.757329 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.757246    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"]
Oct 29 23:01:27.766145 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.763242    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.789239 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.789205    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9d6d4e-openshift-default-dockercfg-km5wx\""
Oct 29 23:01:27.793682 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.793648    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b302c38f-92a7-4243-868b-3c3a04798313-istio-token\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.793877 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.793751    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-workload-certs\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.793877 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.793864    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5kkl\" (UniqueName: \"kubernetes.io/projected/b302c38f-92a7-4243-868b-3c3a04798313-kube-api-access-j5kkl\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.794095 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.793903    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-workload-socket\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.794095 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.793989    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-credential-socket\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.794095 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.794032    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b302c38f-92a7-4243-868b-3c3a04798313-istiod-ca-cert\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.794095 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.794059    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b302c38f-92a7-4243-868b-3c3a04798313-istio-podinfo\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.794308 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.794125    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-istio-data\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.794308 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.794169    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-istio-envoy\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.796312 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.796285    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"]
Oct 29 23:01:27.903835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.896166    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-workload-socket\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.903835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.896284    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-credential-socket\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.903835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.896338    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b302c38f-92a7-4243-868b-3c3a04798313-istiod-ca-cert\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.903835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.896366    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b302c38f-92a7-4243-868b-3c3a04798313-istio-podinfo\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.903835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.896431    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-istio-data\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.903835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.896474    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-istio-envoy\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.903835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.896568    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b302c38f-92a7-4243-868b-3c3a04798313-istio-token\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.903835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.896640    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-workload-certs\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.903835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.896737    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5kkl\" (UniqueName: \"kubernetes.io/projected/b302c38f-92a7-4243-868b-3c3a04798313-kube-api-access-j5kkl\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.903835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.897562    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-workload-socket\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.903835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.897836    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-credential-socket\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.903835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.898416    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b302c38f-92a7-4243-868b-3c3a04798313-istiod-ca-cert\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.903835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.899863    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-istio-data\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.903835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.900118    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-workload-certs\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.904702 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.904365    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-istio-envoy\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.911372 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.911313    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b302c38f-92a7-4243-868b-3c3a04798313-istio-podinfo\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.917472 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.917441    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5kkl\" (UniqueName: \"kubernetes.io/projected/b302c38f-92a7-4243-868b-3c3a04798313-kube-api-access-j5kkl\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:27.921227 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:27.921199    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b302c38f-92a7-4243-868b-3c3a04798313-istio-token\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:28.101346 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:28.101213    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:28.198333 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:28.191880    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"
Oct 29 23:01:28.198333 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:28.192867    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg" event={"ID":"daabb28c-a3d4-43f5-85d7-b09141c8a4fd","Type":"ContainerDied","Data":"31531d637c4551224ce194f140463dcecec8eb7043a31e3dee43380fde4ab502"}
Oct 29 23:01:28.198333 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:28.192946    4559 scope.go:122] "RemoveContainer" containerID="91b6aac3f7c570c7478709b1d31627e07a983b81a8d2f500ab49ba84da2c7a1b"
Oct 29 23:01:28.198333 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:28.193164    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"
Oct 29 23:01:28.290613 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:28.287125    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"]
Oct 29 23:01:28.315278 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:28.315244    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"]
Oct 29 23:01:28.319263 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:28.319230    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/premium-w4-08ebee-kserve-7b64956b4f-8x224"]
Oct 29 23:01:28.357431 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:28.357191    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"]
Oct 29 23:01:28.369958 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:28.365083    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/distinct-2-w4-08ebee-kserve-657686d8c-82jmg"]
Oct 29 23:01:28.655740 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:28.655705    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"]
Oct 29 23:01:28.657180 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:01:28.657149    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb302c38f_92a7_4243_868b_3c3a04798313.slice/crio-a9a12584d88ce06720eccb96afa92795d24fadc6627812651a368f4631905d2c WatchSource:0}: Error finding container a9a12584d88ce06720eccb96afa92795d24fadc6627812651a368f4631905d2c: Status 404 returned error can't find the container with id a9a12584d88ce06720eccb96afa92795d24fadc6627812651a368f4631905d2c
Oct 29 23:01:28.660851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:28.660794    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:01:28.660998 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:28.660891    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:01:28.660998 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:28.660924    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:01:29.203239 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:29.203186    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl" event={"ID":"b302c38f-92a7-4243-868b-3c3a04798313","Type":"ContainerStarted","Data":"acc784ea146aaa70d158583b457a8fe15e243934948b567fb23c07c92b8c8c42"}
Oct 29 23:01:29.203770 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:29.203264    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl" event={"ID":"b302c38f-92a7-4243-868b-3c3a04798313","Type":"ContainerStarted","Data":"a9a12584d88ce06720eccb96afa92795d24fadc6627812651a368f4631905d2c"}
Oct 29 23:01:29.206735 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:29.206702    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7" event={"ID":"732da306-43ba-4e56-a01f-70532d0211cb","Type":"ContainerStarted","Data":"f7d3ced85cd99fc7c2f0f8666acab2f781945056d89a19aa941a766596a44d82"}
Oct 29 23:01:29.206934 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:29.206754    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7" event={"ID":"732da306-43ba-4e56-a01f-70532d0211cb","Type":"ContainerStarted","Data":"5566170a496914a39167e47fe743bab4841da1a1f0ded7336e23c0b628d0534c"}
Oct 29 23:01:29.207090 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:29.207069    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"
Oct 29 23:01:29.229240 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:29.229203    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="779ec9d1-52de-4f60-9f5e-4be62ceb311f" path="/var/lib/kubelet/pods/779ec9d1-52de-4f60-9f5e-4be62ceb311f/volumes"
Oct 29 23:01:29.229776 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:29.229753    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daabb28c-a3d4-43f5-85d7-b09141c8a4fd" path="/var/lib/kubelet/pods/daabb28c-a3d4-43f5-85d7-b09141c8a4fd/volumes"
Oct 29 23:01:29.271724 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:29.271666    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl" podStartSLOduration=2.271645965 podStartE2EDuration="2.271645965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:29.255849501 +0000 UTC m=+2828.167293073" watchObservedRunningTime="2026-10-29 23:01:29.271645965 +0000 UTC m=+2828.183089521"
Oct 29 23:01:29.298783 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:29.298713    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7" podStartSLOduration=2.2986908059999998 podStartE2EDuration="2.298690806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:29.293453085 +0000 UTC m=+2828.204896641" watchObservedRunningTime="2026-10-29 23:01:29.298690806 +0000 UTC m=+2828.210134363"
Oct 29 23:01:30.102841 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:30.102775    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:30.108434 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:30.108405    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:30.215252 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:30.215218    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:30.216342 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:30.216320    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:01:30.335220 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:30.335179    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"]
Oct 29 23:01:30.335485 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:30.335453    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r" podUID="ec19a061-f45f-42f2-aea1-da03fe800597" containerName="istio-proxy" containerID="cri-o://62c13c07b39673a67462946b5db36cd0aa0df1ff4c43270d4f16df3c34f9a823" gracePeriod=30
Oct 29 23:01:30.629004 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:30.628962    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj" podUID="ecd94d51-9423-41c3-8ab2-e94b52ab5904" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:01:30.867231 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:30.867180    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"]
Oct 29 23:01:30.873162 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:30.873134    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"
Oct 29 23:01:30.890269 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:30.889411    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"]
Oct 29 23:01:30.939963 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:30.939916    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:01:30.940146 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:30.939971    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:01:30.969741 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:30.969703    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f3b306-0b68-4152-98c0-960491150159-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv\" (UID: \"c8f3b306-0b68-4152-98c0-960491150159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"
Oct 29 23:01:30.969969 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:30.969942    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f3b306-0b68-4152-98c0-960491150159-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv\" (UID: \"c8f3b306-0b68-4152-98c0-960491150159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"
Oct 29 23:01:30.970016 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:30.969985    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dtp8\" (UniqueName: \"kubernetes.io/projected/c8f3b306-0b68-4152-98c0-960491150159-kube-api-access-2dtp8\") pod \"maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv\" (UID: \"c8f3b306-0b68-4152-98c0-960491150159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"
Oct 29 23:01:31.071832 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.071770    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f3b306-0b68-4152-98c0-960491150159-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv\" (UID: \"c8f3b306-0b68-4152-98c0-960491150159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"
Oct 29 23:01:31.072065 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.071955    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f3b306-0b68-4152-98c0-960491150159-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv\" (UID: \"c8f3b306-0b68-4152-98c0-960491150159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"
Oct 29 23:01:31.072065 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.071987    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtp8\" (UniqueName: \"kubernetes.io/projected/c8f3b306-0b68-4152-98c0-960491150159-kube-api-access-2dtp8\") pod \"maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv\" (UID: \"c8f3b306-0b68-4152-98c0-960491150159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"
Oct 29 23:01:31.072870 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.072794    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f3b306-0b68-4152-98c0-960491150159-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv\" (UID: \"c8f3b306-0b68-4152-98c0-960491150159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"
Oct 29 23:01:31.074956 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.074930    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f3b306-0b68-4152-98c0-960491150159-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv\" (UID: \"c8f3b306-0b68-4152-98c0-960491150159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"
Oct 29 23:01:31.091635 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.091606    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dtp8\" (UniqueName: \"kubernetes.io/projected/c8f3b306-0b68-4152-98c0-960491150159-kube-api-access-2dtp8\") pod \"maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv\" (UID: \"c8f3b306-0b68-4152-98c0-960491150159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"
Oct 29 23:01:31.194270 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.194230    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"
Oct 29 23:01:31.489627 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.489533    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"]
Oct 29 23:01:31.495846 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.495787    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"
Oct 29 23:01:31.506628 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:01:31.506597    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f3b306_0b68_4152_98c0_960491150159.slice/crio-fa4e56ffc4c80aaf8b9e3a76542c383697080ab80c21e2a4680961cc3661bbac WatchSource:0}: Error finding container fa4e56ffc4c80aaf8b9e3a76542c383697080ab80c21e2a4680961cc3661bbac: Status 404 returned error can't find the container with id fa4e56ffc4c80aaf8b9e3a76542c383697080ab80c21e2a4680961cc3661bbac
Oct 29 23:01:31.506628 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.506622    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"]
Oct 29 23:01:31.517872 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.517843    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"]
Oct 29 23:01:31.579683 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.579644    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4940d541-edda-4219-8e75-2d3702b5c3c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f6b42db-rq9l2\" (UID: \"4940d541-edda-4219-8e75-2d3702b5c3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"
Oct 29 23:01:31.579920 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.579901    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4940d541-edda-4219-8e75-2d3702b5c3c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f6b42db-rq9l2\" (UID: \"4940d541-edda-4219-8e75-2d3702b5c3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"
Oct 29 23:01:31.580007 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.579940    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghbxl\" (UniqueName: \"kubernetes.io/projected/4940d541-edda-4219-8e75-2d3702b5c3c6-kube-api-access-ghbxl\") pod \"maas-api-revoke-sub-4f6b42db-rq9l2\" (UID: \"4940d541-edda-4219-8e75-2d3702b5c3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"
Oct 29 23:01:31.681288 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.681250    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4940d541-edda-4219-8e75-2d3702b5c3c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f6b42db-rq9l2\" (UID: \"4940d541-edda-4219-8e75-2d3702b5c3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"
Oct 29 23:01:31.681487 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.681301    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghbxl\" (UniqueName: \"kubernetes.io/projected/4940d541-edda-4219-8e75-2d3702b5c3c6-kube-api-access-ghbxl\") pod \"maas-api-revoke-sub-4f6b42db-rq9l2\" (UID: \"4940d541-edda-4219-8e75-2d3702b5c3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"
Oct 29 23:01:31.681487 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.681379    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4940d541-edda-4219-8e75-2d3702b5c3c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f6b42db-rq9l2\" (UID: \"4940d541-edda-4219-8e75-2d3702b5c3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"
Oct 29 23:01:31.682046 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.682019    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4940d541-edda-4219-8e75-2d3702b5c3c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f6b42db-rq9l2\" (UID: \"4940d541-edda-4219-8e75-2d3702b5c3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"
Oct 29 23:01:31.684026 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.683996    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4940d541-edda-4219-8e75-2d3702b5c3c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f6b42db-rq9l2\" (UID: \"4940d541-edda-4219-8e75-2d3702b5c3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"
Oct 29 23:01:31.702382 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.702351    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghbxl\" (UniqueName: \"kubernetes.io/projected/4940d541-edda-4219-8e75-2d3702b5c3c6-kube-api-access-ghbxl\") pod \"maas-api-revoke-sub-4f6b42db-rq9l2\" (UID: \"4940d541-edda-4219-8e75-2d3702b5c3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"
Oct 29 23:01:31.850577 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.850533    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"
Oct 29 23:01:31.974736 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:31.974698    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:01:32.274605 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:32.273008    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv" event={"ID":"c8f3b306-0b68-4152-98c0-960491150159","Type":"ContainerStarted","Data":"79941e7d25cfa124822e889ae9960b5dd22b2558af929d5cd8c3249428a88b73"}
Oct 29 23:01:32.274605 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:32.273068    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv" event={"ID":"c8f3b306-0b68-4152-98c0-960491150159","Type":"ContainerStarted","Data":"fa4e56ffc4c80aaf8b9e3a76542c383697080ab80c21e2a4680961cc3661bbac"}
Oct 29 23:01:32.274605 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:32.273352    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"]
Oct 29 23:01:32.338304 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:32.338257    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv" podStartSLOduration=2.3382411579999998 podStartE2EDuration="2.338241158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:32.332249656 +0000 UTC m=+2831.243693211" watchObservedRunningTime="2026-10-29 23:01:32.338241158 +0000 UTC m=+2831.249684711"
Oct 29 23:01:33.126310 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:33.126258    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"]
Oct 29 23:01:33.126695 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:33.126565    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt" podUID="2c3db4b3-0ba7-49d3-9349-327aa6022a71" containerName="istio-proxy" containerID="cri-o://71b77cbf4daed37016865e2287e649cb31c1ae31504f884f42a229b6c400505e" gracePeriod=30
Oct 29 23:01:33.392899 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:33.392442    4559 generic.go:356] "Generic (PLEG): container finished" podID="c8f3b306-0b68-4152-98c0-960491150159" containerID="79941e7d25cfa124822e889ae9960b5dd22b2558af929d5cd8c3249428a88b73" exitCode=0
Oct 29 23:01:33.397894 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:33.397575    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2" event={"ID":"4940d541-edda-4219-8e75-2d3702b5c3c6","Type":"ContainerStarted","Data":"4003f9aa213aaf8157da23fe6b9f704db63742c3f7bb9b44674e23ff183496b5"}
Oct 29 23:01:33.397894 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:33.397655    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2" event={"ID":"4940d541-edda-4219-8e75-2d3702b5c3c6","Type":"ContainerStarted","Data":"a1b2469760fd9fe0ca1716a9b27e0b29c2bc24ce4764bbc05de6442c86e69824"}
Oct 29 23:01:33.397894 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:33.397686    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv" event={"ID":"c8f3b306-0b68-4152-98c0-960491150159","Type":"ContainerDied","Data":"79941e7d25cfa124822e889ae9960b5dd22b2558af929d5cd8c3249428a88b73"}
Oct 29 23:01:33.433835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:33.433260    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2" podStartSLOduration=2.433245481 podStartE2EDuration="2.433245481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:33.431540537 +0000 UTC m=+2832.342984091" watchObservedRunningTime="2026-10-29 23:01:33.433245481 +0000 UTC m=+2832.344689032"
Oct 29 23:01:33.693649 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:33.693568    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:01:33.693934 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:33.693893    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt" podUID="2c3db4b3-0ba7-49d3-9349-327aa6022a71" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:01:34.403484 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:34.403446    4559 generic.go:356] "Generic (PLEG): container finished" podID="4940d541-edda-4219-8e75-2d3702b5c3c6" containerID="4003f9aa213aaf8157da23fe6b9f704db63742c3f7bb9b44674e23ff183496b5" exitCode=0
Oct 29 23:01:34.403986 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:34.403541    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2" event={"ID":"4940d541-edda-4219-8e75-2d3702b5c3c6","Type":"ContainerDied","Data":"4003f9aa213aaf8157da23fe6b9f704db63742c3f7bb9b44674e23ff183496b5"}
Oct 29 23:01:34.601325 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:34.601297    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"
Oct 29 23:01:34.692616 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:34.692513    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f3b306-0b68-4152-98c0-960491150159-kube-api-access-2dtp8\" (UniqueName: \"kubernetes.io/projected/c8f3b306-0b68-4152-98c0-960491150159-kube-api-access-2dtp8\") pod \"c8f3b306-0b68-4152-98c0-960491150159\" (UID: \"c8f3b306-0b68-4152-98c0-960491150159\") "
Oct 29 23:01:34.692801 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:34.692781    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8f3b306-0b68-4152-98c0-960491150159-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f3b306-0b68-4152-98c0-960491150159-maas-api-xks-ca\") pod \"c8f3b306-0b68-4152-98c0-960491150159\" (UID: \"c8f3b306-0b68-4152-98c0-960491150159\") "
Oct 29 23:01:34.692930 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:34.692908    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f3b306-0b68-4152-98c0-960491150159-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f3b306-0b68-4152-98c0-960491150159-maas-api-openshift-ca\") pod \"c8f3b306-0b68-4152-98c0-960491150159\" (UID: \"c8f3b306-0b68-4152-98c0-960491150159\") "
Oct 29 23:01:34.693475 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:34.693399    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f3b306-0b68-4152-98c0-960491150159-maas-api-openshift-ca" pod "c8f3b306-0b68-4152-98c0-960491150159" (UID: "c8f3b306-0b68-4152-98c0-960491150159"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:34.693610 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:34.693570    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f3b306-0b68-4152-98c0-960491150159-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:34.695699 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:34.695660    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f3b306-0b68-4152-98c0-960491150159-maas-api-xks-ca" pod "c8f3b306-0b68-4152-98c0-960491150159" (UID: "c8f3b306-0b68-4152-98c0-960491150159"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:34.695699 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:34.695666    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f3b306-0b68-4152-98c0-960491150159-kube-api-access-2dtp8" pod "c8f3b306-0b68-4152-98c0-960491150159" (UID: "c8f3b306-0b68-4152-98c0-960491150159"). InnerVolumeSpecName "kube-api-access-2dtp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:34.795126 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:34.795087    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f3b306-0b68-4152-98c0-960491150159-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:34.795126 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:34.795127    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dtp8\" (UniqueName: \"kubernetes.io/projected/c8f3b306-0b68-4152-98c0-960491150159-kube-api-access-2dtp8\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:35.412461 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.412435    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"
Oct 29 23:01:35.412929 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.412428    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv" event={"ID":"c8f3b306-0b68-4152-98c0-960491150159","Type":"ContainerDied","Data":"fa4e56ffc4c80aaf8b9e3a76542c383697080ab80c21e2a4680961cc3661bbac"}
Oct 29 23:01:35.412929 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.412573    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa4e56ffc4c80aaf8b9e3a76542c383697080ab80c21e2a4680961cc3661bbac"
Oct 29 23:01:35.617773 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.617741    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"
Oct 29 23:01:35.664260 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.664232    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 23:01:35.752959 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.724520    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4940d541-edda-4219-8e75-2d3702b5c3c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4940d541-edda-4219-8e75-2d3702b5c3c6-maas-api-openshift-ca\") pod \"4940d541-edda-4219-8e75-2d3702b5c3c6\" (UID: \"4940d541-edda-4219-8e75-2d3702b5c3c6\") "
Oct 29 23:01:35.752959 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.724588    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4940d541-edda-4219-8e75-2d3702b5c3c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4940d541-edda-4219-8e75-2d3702b5c3c6-maas-api-xks-ca\") pod \"4940d541-edda-4219-8e75-2d3702b5c3c6\" (UID: \"4940d541-edda-4219-8e75-2d3702b5c3c6\") "
Oct 29 23:01:35.752959 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.724854    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4940d541-edda-4219-8e75-2d3702b5c3c6-kube-api-access-ghbxl\" (UniqueName: \"kubernetes.io/projected/4940d541-edda-4219-8e75-2d3702b5c3c6-kube-api-access-ghbxl\") pod \"4940d541-edda-4219-8e75-2d3702b5c3c6\" (UID: \"4940d541-edda-4219-8e75-2d3702b5c3c6\") "
Oct 29 23:01:35.756330 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.754786    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4940d541-edda-4219-8e75-2d3702b5c3c6-maas-api-openshift-ca" pod "4940d541-edda-4219-8e75-2d3702b5c3c6" (UID: "4940d541-edda-4219-8e75-2d3702b5c3c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:35.770871 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.764775    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4940d541-edda-4219-8e75-2d3702b5c3c6-kube-api-access-ghbxl" pod "4940d541-edda-4219-8e75-2d3702b5c3c6" (UID: "4940d541-edda-4219-8e75-2d3702b5c3c6"). InnerVolumeSpecName "kube-api-access-ghbxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:35.774353 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.774318    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4940d541-edda-4219-8e75-2d3702b5c3c6-maas-api-xks-ca" pod "4940d541-edda-4219-8e75-2d3702b5c3c6" (UID: "4940d541-edda-4219-8e75-2d3702b5c3c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:35.826642 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.826542    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-workload-certs\") pod \"ec19a061-f45f-42f2-aea1-da03fe800597\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") "
Oct 29 23:01:35.826642 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.826594    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec19a061-f45f-42f2-aea1-da03fe800597-istio-token\" (UniqueName: \"kubernetes.io/projected/ec19a061-f45f-42f2-aea1-da03fe800597-istio-token\") pod \"ec19a061-f45f-42f2-aea1-da03fe800597\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") "
Oct 29 23:01:35.826957 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.826706    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-credential-socket\") pod \"ec19a061-f45f-42f2-aea1-da03fe800597\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") "
Oct 29 23:01:35.826957 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.826763    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-istio-data\") pod \"ec19a061-f45f-42f2-aea1-da03fe800597\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") "
Oct 29 23:01:35.826957 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.826797    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec19a061-f45f-42f2-aea1-da03fe800597-kube-api-access-24g72\" (UniqueName: \"kubernetes.io/projected/ec19a061-f45f-42f2-aea1-da03fe800597-kube-api-access-24g72\") pod \"ec19a061-f45f-42f2-aea1-da03fe800597\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") "
Oct 29 23:01:35.826957 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.826889    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-workload-certs" pod "ec19a061-f45f-42f2-aea1-da03fe800597" (UID: "ec19a061-f45f-42f2-aea1-da03fe800597"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:35.827160 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.826969    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-istio-envoy\") pod \"ec19a061-f45f-42f2-aea1-da03fe800597\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") "
Oct 29 23:01:35.827160 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.827035    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-workload-socket\") pod \"ec19a061-f45f-42f2-aea1-da03fe800597\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") "
Oct 29 23:01:35.827160 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.827051    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-credential-socket" pod "ec19a061-f45f-42f2-aea1-da03fe800597" (UID: "ec19a061-f45f-42f2-aea1-da03fe800597"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:35.827160 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.827072    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec19a061-f45f-42f2-aea1-da03fe800597-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec19a061-f45f-42f2-aea1-da03fe800597-istio-podinfo\") pod \"ec19a061-f45f-42f2-aea1-da03fe800597\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") "
Oct 29 23:01:35.827160 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.827069    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-istio-data" pod "ec19a061-f45f-42f2-aea1-da03fe800597" (UID: "ec19a061-f45f-42f2-aea1-da03fe800597"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:35.827160 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.827159    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec19a061-f45f-42f2-aea1-da03fe800597-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec19a061-f45f-42f2-aea1-da03fe800597-istiod-ca-cert\") pod \"ec19a061-f45f-42f2-aea1-da03fe800597\" (UID: \"ec19a061-f45f-42f2-aea1-da03fe800597\") "
Oct 29 23:01:35.827487 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.827272    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-workload-socket" pod "ec19a061-f45f-42f2-aea1-da03fe800597" (UID: "ec19a061-f45f-42f2-aea1-da03fe800597"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:35.827581 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.827557    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec19a061-f45f-42f2-aea1-da03fe800597-istiod-ca-cert" pod "ec19a061-f45f-42f2-aea1-da03fe800597" (UID: "ec19a061-f45f-42f2-aea1-da03fe800597"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:35.827939 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.827909    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:35.827939 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.827930    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:35.827939 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.827947    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghbxl\" (UniqueName: \"kubernetes.io/projected/4940d541-edda-4219-8e75-2d3702b5c3c6-kube-api-access-ghbxl\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:35.827939 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.827961    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:35.827939 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.827974    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec19a061-f45f-42f2-aea1-da03fe800597-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:35.827939 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.827989    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4940d541-edda-4219-8e75-2d3702b5c3c6-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:35.828267 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.828002    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4940d541-edda-4219-8e75-2d3702b5c3c6-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:35.828267 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.828014    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:35.829182 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.829163    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec19a061-f45f-42f2-aea1-da03fe800597-istio-token" pod "ec19a061-f45f-42f2-aea1-da03fe800597" (UID: "ec19a061-f45f-42f2-aea1-da03fe800597"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:35.829382 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.829367    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-istio-envoy" pod "ec19a061-f45f-42f2-aea1-da03fe800597" (UID: "ec19a061-f45f-42f2-aea1-da03fe800597"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:35.829800 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.829785    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec19a061-f45f-42f2-aea1-da03fe800597-istio-podinfo" pod "ec19a061-f45f-42f2-aea1-da03fe800597" (UID: "ec19a061-f45f-42f2-aea1-da03fe800597"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:35.829883 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.829862    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec19a061-f45f-42f2-aea1-da03fe800597-kube-api-access-24g72" pod "ec19a061-f45f-42f2-aea1-da03fe800597" (UID: "ec19a061-f45f-42f2-aea1-da03fe800597"). InnerVolumeSpecName "kube-api-access-24g72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:35.929953 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.929920    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec19a061-f45f-42f2-aea1-da03fe800597-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:35.929953 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.929951    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec19a061-f45f-42f2-aea1-da03fe800597-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:35.929953 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.929961    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24g72\" (UniqueName: \"kubernetes.io/projected/ec19a061-f45f-42f2-aea1-da03fe800597-kube-api-access-24g72\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:35.930174 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:35.929971    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec19a061-f45f-42f2-aea1-da03fe800597-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:36.422183 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:36.422154    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"
Oct 29 23:01:36.422627 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:36.422195    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2" event={"ID":"4940d541-edda-4219-8e75-2d3702b5c3c6","Type":"ContainerDied","Data":"a1b2469760fd9fe0ca1716a9b27e0b29c2bc24ce4764bbc05de6442c86e69824"}
Oct 29 23:01:36.422627 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:36.422243    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1b2469760fd9fe0ca1716a9b27e0b29c2bc24ce4764bbc05de6442c86e69824"
Oct 29 23:01:36.424227 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:36.424201    4559 generic.go:356] "Generic (PLEG): container finished" podID="ec19a061-f45f-42f2-aea1-da03fe800597" containerID="62c13c07b39673a67462946b5db36cd0aa0df1ff4c43270d4f16df3c34f9a823" exitCode=0
Oct 29 23:01:36.424384 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:36.424238    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r" event={"ID":"ec19a061-f45f-42f2-aea1-da03fe800597","Type":"ContainerDied","Data":"62c13c07b39673a67462946b5db36cd0aa0df1ff4c43270d4f16df3c34f9a823"}
Oct 29 23:01:36.424384 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:36.424252    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"
Oct 29 23:01:36.424384 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:36.424284    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r" event={"ID":"ec19a061-f45f-42f2-aea1-da03fe800597","Type":"ContainerDied","Data":"d8a3e0b49593ee777b07c3122cb572606479932e87bfeba68e35607680f8471d"}
Oct 29 23:01:36.424384 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:36.424313    4559 scope.go:122] "RemoveContainer" containerID="62c13c07b39673a67462946b5db36cd0aa0df1ff4c43270d4f16df3c34f9a823"
Oct 29 23:01:36.441908 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:36.441880    4559 scope.go:122] "RemoveContainer" containerID="305c5988baf041f6f2d0a220e8df32aec6982d2b2cbd1fce10400f2ed240bebb"
Oct 29 23:01:36.453468 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:36.453441    4559 scope.go:122] "RemoveContainer" containerID="62c13c07b39673a67462946b5db36cd0aa0df1ff4c43270d4f16df3c34f9a823"
Oct 29 23:01:36.453902 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:01:36.453875    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c13c07b39673a67462946b5db36cd0aa0df1ff4c43270d4f16df3c34f9a823\": container with ID starting with 62c13c07b39673a67462946b5db36cd0aa0df1ff4c43270d4f16df3c34f9a823 not found: ID does not exist" containerID="62c13c07b39673a67462946b5db36cd0aa0df1ff4c43270d4f16df3c34f9a823"
Oct 29 23:01:36.454018 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:36.453937    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c13c07b39673a67462946b5db36cd0aa0df1ff4c43270d4f16df3c34f9a823"} err="failed to get container status \"62c13c07b39673a67462946b5db36cd0aa0df1ff4c43270d4f16df3c34f9a823\": rpc error: code = NotFound desc = could not find container \"62c13c07b39673a67462946b5db36cd0aa0df1ff4c43270d4f16df3c34f9a823\": container with ID starting with 62c13c07b39673a67462946b5db36cd0aa0df1ff4c43270d4f16df3c34f9a823 not found: ID does not exist"
Oct 29 23:01:36.454018 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:36.453964    4559 scope.go:122] "RemoveContainer" containerID="305c5988baf041f6f2d0a220e8df32aec6982d2b2cbd1fce10400f2ed240bebb"
Oct 29 23:01:36.454332 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:01:36.454310    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305c5988baf041f6f2d0a220e8df32aec6982d2b2cbd1fce10400f2ed240bebb\": container with ID starting with 305c5988baf041f6f2d0a220e8df32aec6982d2b2cbd1fce10400f2ed240bebb not found: ID does not exist" containerID="305c5988baf041f6f2d0a220e8df32aec6982d2b2cbd1fce10400f2ed240bebb"
Oct 29 23:01:36.454449 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:36.454336    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305c5988baf041f6f2d0a220e8df32aec6982d2b2cbd1fce10400f2ed240bebb"} err="failed to get container status \"305c5988baf041f6f2d0a220e8df32aec6982d2b2cbd1fce10400f2ed240bebb\": rpc error: code = NotFound desc = could not find container \"305c5988baf041f6f2d0a220e8df32aec6982d2b2cbd1fce10400f2ed240bebb\": container with ID starting with 305c5988baf041f6f2d0a220e8df32aec6982d2b2cbd1fce10400f2ed240bebb not found: ID does not exist"
Oct 29 23:01:36.510669 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:36.510633    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"]
Oct 29 23:01:36.522955 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:36.522927    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-83f120-openshift-default-78b78cf899-x822r"]
Oct 29 23:01:37.230963 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:37.230928    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec19a061-f45f-42f2-aea1-da03fe800597" path="/var/lib/kubelet/pods/ec19a061-f45f-42f2-aea1-da03fe800597/volumes"
Oct 29 23:01:38.450861 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.450696    4559 generic.go:356] "Generic (PLEG): container finished" podID="2c3db4b3-0ba7-49d3-9349-327aa6022a71" containerID="71b77cbf4daed37016865e2287e649cb31c1ae31504f884f42a229b6c400505e" exitCode=0
Oct 29 23:01:38.450861 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.450775    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt" event={"ID":"2c3db4b3-0ba7-49d3-9349-327aa6022a71","Type":"ContainerDied","Data":"71b77cbf4daed37016865e2287e649cb31c1ae31504f884f42a229b6c400505e"}
Oct 29 23:01:38.451441 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.450902    4559 scope.go:122] "RemoveContainer" containerID="2213aca4d64dab8d98d60076dc33ffa470aba170197d61a8e08e169e526f13ff"
Oct 29 23:01:38.503508 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.503476    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 23:01:38.585675 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.585620    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-data\") pod \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") "
Oct 29 23:01:38.585978 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.585697    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istiod-ca-cert\") pod \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") "
Oct 29 23:01:38.585978 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.585866    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-workload-certs\") pod \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") "
Oct 29 23:01:38.585978 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.585901    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-workload-socket\") pod \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") "
Oct 29 23:01:38.585978 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.585952    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-credential-socket\") pod \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") "
Oct 29 23:01:38.586253 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.586037    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-envoy\") pod \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") "
Oct 29 23:01:38.586253 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.586008    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-data" pod "2c3db4b3-0ba7-49d3-9349-327aa6022a71" (UID: "2c3db4b3-0ba7-49d3-9349-327aa6022a71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:38.586253 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.586134    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-podinfo\") pod \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") "
Oct 29 23:01:38.586253 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.586232    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-token\" (UniqueName: \"kubernetes.io/projected/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-token\") pod \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") "
Oct 29 23:01:38.586482 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.586265    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3db4b3-0ba7-49d3-9349-327aa6022a71-kube-api-access-w8gr4\" (UniqueName: \"kubernetes.io/projected/2c3db4b3-0ba7-49d3-9349-327aa6022a71-kube-api-access-w8gr4\") pod \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\" (UID: \"2c3db4b3-0ba7-49d3-9349-327aa6022a71\") "
Oct 29 23:01:38.586482 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.586298    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-workload-socket" pod "2c3db4b3-0ba7-49d3-9349-327aa6022a71" (UID: "2c3db4b3-0ba7-49d3-9349-327aa6022a71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:38.586482 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.586322    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istiod-ca-cert" pod "2c3db4b3-0ba7-49d3-9349-327aa6022a71" (UID: "2c3db4b3-0ba7-49d3-9349-327aa6022a71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:38.586650 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.586487    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-workload-certs" pod "2c3db4b3-0ba7-49d3-9349-327aa6022a71" (UID: "2c3db4b3-0ba7-49d3-9349-327aa6022a71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:38.586960 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.586932    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:38.587064 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.586964    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:38.587064 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.586979    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:38.587064 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.586992    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:38.587240 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.587201    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-credential-socket" pod "2c3db4b3-0ba7-49d3-9349-327aa6022a71" (UID: "2c3db4b3-0ba7-49d3-9349-327aa6022a71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:38.589913 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.589868    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-envoy" pod "2c3db4b3-0ba7-49d3-9349-327aa6022a71" (UID: "2c3db4b3-0ba7-49d3-9349-327aa6022a71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:38.589913 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.589886    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-podinfo" pod "2c3db4b3-0ba7-49d3-9349-327aa6022a71" (UID: "2c3db4b3-0ba7-49d3-9349-327aa6022a71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:38.590169 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.590148    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-token" pod "2c3db4b3-0ba7-49d3-9349-327aa6022a71" (UID: "2c3db4b3-0ba7-49d3-9349-327aa6022a71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:38.590233 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.590181    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3db4b3-0ba7-49d3-9349-327aa6022a71-kube-api-access-w8gr4" pod "2c3db4b3-0ba7-49d3-9349-327aa6022a71" (UID: "2c3db4b3-0ba7-49d3-9349-327aa6022a71"). InnerVolumeSpecName "kube-api-access-w8gr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:38.688622 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.688576    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:38.688622 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.688615    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8gr4\" (UniqueName: \"kubernetes.io/projected/2c3db4b3-0ba7-49d3-9349-327aa6022a71-kube-api-access-w8gr4\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:38.688622 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.688624    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:38.688622 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.688633    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:38.688982 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:38.688642    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3db4b3-0ba7-49d3-9349-327aa6022a71-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:39.462851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:39.462774    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"
Oct 29 23:01:39.462851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:39.462831    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt" event={"ID":"2c3db4b3-0ba7-49d3-9349-327aa6022a71","Type":"ContainerDied","Data":"674c6621e68efbddc56794fa4120597f7bcc43cf4517eded5f76b1aca61271ac"}
Oct 29 23:01:39.463446 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:39.462905    4559 scope.go:122] "RemoveContainer" containerID="71b77cbf4daed37016865e2287e649cb31c1ae31504f884f42a229b6c400505e"
Oct 29 23:01:39.525682 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:39.525588    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"]
Oct 29 23:01:39.541854 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:39.540656    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b8f54c-openshift-default-69ccbf5c6c-mwkmt"]
Oct 29 23:01:40.034853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.021884    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"]
Oct 29 23:01:40.034853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.022888    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2c3db4b3-0ba7-49d3-9349-327aa6022a71" containerName="istio-proxy"
Oct 29 23:01:40.034853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.022910    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3db4b3-0ba7-49d3-9349-327aa6022a71" containerName="istio-proxy"
Oct 29 23:01:40.034853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.022932    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4940d541-edda-4219-8e75-2d3702b5c3c6" containerName="revoke-keys"
Oct 29 23:01:40.034853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.022939    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4940d541-edda-4219-8e75-2d3702b5c3c6" containerName="revoke-keys"
Oct 29 23:01:40.034853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.022953    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ec19a061-f45f-42f2-aea1-da03fe800597" containerName="istio-proxy"
Oct 29 23:01:40.034853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.022959    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec19a061-f45f-42f2-aea1-da03fe800597" containerName="istio-proxy"
Oct 29 23:01:40.034853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.022975    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c8f3b306-0b68-4152-98c0-960491150159" containerName="revoke-keys"
Oct 29 23:01:40.034853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.022982    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f3b306-0b68-4152-98c0-960491150159" containerName="revoke-keys"
Oct 29 23:01:40.034853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.023025    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2c3db4b3-0ba7-49d3-9349-327aa6022a71" containerName="istio-proxy"
Oct 29 23:01:40.034853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.023032    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3db4b3-0ba7-49d3-9349-327aa6022a71" containerName="istio-proxy"
Oct 29 23:01:40.034853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.023165    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4940d541-edda-4219-8e75-2d3702b5c3c6" containerName="revoke-keys"
Oct 29 23:01:40.034853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.023175    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec19a061-f45f-42f2-aea1-da03fe800597" containerName="istio-proxy"
Oct 29 23:01:40.034853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.023187    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f3b306-0b68-4152-98c0-960491150159" containerName="revoke-keys"
Oct 29 23:01:40.034853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.023201    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3db4b3-0ba7-49d3-9349-327aa6022a71" containerName="istio-proxy"
Oct 29 23:01:40.034853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.023213    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3db4b3-0ba7-49d3-9349-327aa6022a71" containerName="istio-proxy"
Oct 29 23:01:40.051869 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.050102    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"
Oct 29 23:01:40.067676 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.067612    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-63e89c\""
Oct 29 23:01:40.067901 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.067710    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-63e89c-dockercfg-ctnmm\""
Oct 29 23:01:40.132378 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.132331    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28022dfc-ac33-46b0-ba0e-066502f8acd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr\" (UID: \"28022dfc-ac33-46b0-ba0e-066502f8acd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"
Oct 29 23:01:40.132676 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.132652    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l695m\" (UniqueName: \"kubernetes.io/projected/28022dfc-ac33-46b0-ba0e-066502f8acd5-kube-api-access-l695m\") pod \"payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr\" (UID: \"28022dfc-ac33-46b0-ba0e-066502f8acd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"
Oct 29 23:01:40.171974 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.171939    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"]
Oct 29 23:01:40.193492 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.193454    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"]
Oct 29 23:01:40.194684 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.194658    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ec19a061-f45f-42f2-aea1-da03fe800597" containerName="istio-proxy"
Oct 29 23:01:40.194684 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.194686    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec19a061-f45f-42f2-aea1-da03fe800597" containerName="istio-proxy"
Oct 29 23:01:40.194919 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.194902    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec19a061-f45f-42f2-aea1-da03fe800597" containerName="istio-proxy"
Oct 29 23:01:40.199712 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.199688    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"
Oct 29 23:01:40.209763 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.209733    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"]
Oct 29 23:01:40.218555 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.218526    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"
Oct 29 23:01:40.233827 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.233774    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbc13347-02e3-42dd-88f6-bf1503c428ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv\" (UID: \"dbc13347-02e3-42dd-88f6-bf1503c428ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"
Oct 29 23:01:40.234024 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.234007    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l695m\" (UniqueName: \"kubernetes.io/projected/28022dfc-ac33-46b0-ba0e-066502f8acd5-kube-api-access-l695m\") pod \"payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr\" (UID: \"28022dfc-ac33-46b0-ba0e-066502f8acd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"
Oct 29 23:01:40.234160 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.234060    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5ms8\" (UniqueName: \"kubernetes.io/projected/dbc13347-02e3-42dd-88f6-bf1503c428ef-kube-api-access-g5ms8\") pod \"payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv\" (UID: \"dbc13347-02e3-42dd-88f6-bf1503c428ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"
Oct 29 23:01:40.234222 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.234155    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28022dfc-ac33-46b0-ba0e-066502f8acd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr\" (UID: \"28022dfc-ac33-46b0-ba0e-066502f8acd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"
Oct 29 23:01:40.234782 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.234758    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28022dfc-ac33-46b0-ba0e-066502f8acd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr\" (UID: \"28022dfc-ac33-46b0-ba0e-066502f8acd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"
Oct 29 23:01:40.253736 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.253704    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l695m\" (UniqueName: \"kubernetes.io/projected/28022dfc-ac33-46b0-ba0e-066502f8acd5-kube-api-access-l695m\") pod \"payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr\" (UID: \"28022dfc-ac33-46b0-ba0e-066502f8acd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"
Oct 29 23:01:40.335679 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.335574    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbc13347-02e3-42dd-88f6-bf1503c428ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv\" (UID: \"dbc13347-02e3-42dd-88f6-bf1503c428ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"
Oct 29 23:01:40.335867 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.335688    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5ms8\" (UniqueName: \"kubernetes.io/projected/dbc13347-02e3-42dd-88f6-bf1503c428ef-kube-api-access-g5ms8\") pod \"payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv\" (UID: \"dbc13347-02e3-42dd-88f6-bf1503c428ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"
Oct 29 23:01:40.336349 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.336326    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbc13347-02e3-42dd-88f6-bf1503c428ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv\" (UID: \"dbc13347-02e3-42dd-88f6-bf1503c428ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"
Oct 29 23:01:40.355028 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.354999    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5ms8\" (UniqueName: \"kubernetes.io/projected/dbc13347-02e3-42dd-88f6-bf1503c428ef-kube-api-access-g5ms8\") pod \"payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv\" (UID: \"dbc13347-02e3-42dd-88f6-bf1503c428ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"
Oct 29 23:01:40.482669 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.482621    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"
Oct 29 23:01:40.525095 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.525025    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"
Oct 29 23:01:40.610920 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.608334    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj" podUID="ecd94d51-9423-41c3-8ab2-e94b52ab5904" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:01:40.791738 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.791704    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"]
Oct 29 23:01:40.792047 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:01:40.792024    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28022dfc_ac33_46b0_ba0e_066502f8acd5.slice/crio-8a4898884c9ef43fae0716b1b70285ff7fe1d5f17aac9fcf70f8b4dbcc20fb78 WatchSource:0}: Error finding container 8a4898884c9ef43fae0716b1b70285ff7fe1d5f17aac9fcf70f8b4dbcc20fb78: Status 404 returned error can't find the container with id 8a4898884c9ef43fae0716b1b70285ff7fe1d5f17aac9fcf70f8b4dbcc20fb78
Oct 29 23:01:40.910694 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:40.906777    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"]
Oct 29 23:01:41.243271 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:41.243180    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c3db4b3-0ba7-49d3-9349-327aa6022a71" path="/var/lib/kubelet/pods/2c3db4b3-0ba7-49d3-9349-327aa6022a71/volumes"
Oct 29 23:01:41.486863 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:41.486801    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv" event={"ID":"dbc13347-02e3-42dd-88f6-bf1503c428ef","Type":"ContainerStarted","Data":"8661bc9a6c87fb9f5a5182d59543a3185c3c2cff22eff6d0fbf649f4996b5512"}
Oct 29 23:01:41.487366 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:41.486890    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"
Oct 29 23:01:41.487366 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:41.486914    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv" event={"ID":"dbc13347-02e3-42dd-88f6-bf1503c428ef","Type":"ContainerStarted","Data":"d1b59a1ec8929c3dbf7c86f8e17ccf0f77c311c7cf5dc75cfd8d60ffc48b9838"}
Oct 29 23:01:41.488639 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:41.488610    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr" event={"ID":"28022dfc-ac33-46b0-ba0e-066502f8acd5","Type":"ContainerStarted","Data":"6868a61a9a84f616291d93c3379c91f0dd118c4ffd82bc5b1feaa8984002521f"}
Oct 29 23:01:41.488803 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:41.488652    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr" event={"ID":"28022dfc-ac33-46b0-ba0e-066502f8acd5","Type":"ContainerStarted","Data":"8a4898884c9ef43fae0716b1b70285ff7fe1d5f17aac9fcf70f8b4dbcc20fb78"}
Oct 29 23:01:41.488803 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:41.488765    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"
Oct 29 23:01:41.525800 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:41.525673    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv" podStartSLOduration=1.525654994 podStartE2EDuration="1.525654994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:41.525158185 +0000 UTC m=+2840.436601736" watchObservedRunningTime="2026-10-29 23:01:41.525654994 +0000 UTC m=+2840.437098549"
Oct 29 23:01:41.568329 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:41.568274    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr" podStartSLOduration=2.568258797 podStartE2EDuration="2.568258797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:41.565284922 +0000 UTC m=+2840.476728502" watchObservedRunningTime="2026-10-29 23:01:41.568258797 +0000 UTC m=+2840.479702350"
Oct 29 23:01:42.533902 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:42.505085    4559 generic.go:356] "Generic (PLEG): container finished" podID="ecd94d51-9423-41c3-8ab2-e94b52ab5904" containerID="536623d37bcad3265f823d0d48ec46bb5096d871c44b055c3811335a8e0ac5fa" exitCode=0
Oct 29 23:01:42.533902 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:42.506262    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj" event={"ID":"ecd94d51-9423-41c3-8ab2-e94b52ab5904","Type":"ContainerDied","Data":"536623d37bcad3265f823d0d48ec46bb5096d871c44b055c3811335a8e0ac5fa"}
Oct 29 23:01:42.627112 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:42.627075    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"
Oct 29 23:01:42.699579 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:42.699533    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd94d51-9423-41c3-8ab2-e94b52ab5904-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd94d51-9423-41c3-8ab2-e94b52ab5904-plugins-config-volume\") pod \"ecd94d51-9423-41c3-8ab2-e94b52ab5904\" (UID: \"ecd94d51-9423-41c3-8ab2-e94b52ab5904\") "
Oct 29 23:01:42.699797 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:42.699689    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd94d51-9423-41c3-8ab2-e94b52ab5904-kube-api-access-7q7kv\" (UniqueName: \"kubernetes.io/projected/ecd94d51-9423-41c3-8ab2-e94b52ab5904-kube-api-access-7q7kv\") pod \"ecd94d51-9423-41c3-8ab2-e94b52ab5904\" (UID: \"ecd94d51-9423-41c3-8ab2-e94b52ab5904\") "
Oct 29 23:01:42.700668 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:42.700637    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd94d51-9423-41c3-8ab2-e94b52ab5904-plugins-config-volume" pod "ecd94d51-9423-41c3-8ab2-e94b52ab5904" (UID: "ecd94d51-9423-41c3-8ab2-e94b52ab5904"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:42.703129 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:42.703099    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd94d51-9423-41c3-8ab2-e94b52ab5904-kube-api-access-7q7kv" pod "ecd94d51-9423-41c3-8ab2-e94b52ab5904" (UID: "ecd94d51-9423-41c3-8ab2-e94b52ab5904"). InnerVolumeSpecName "kube-api-access-7q7kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:42.801527 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:42.801433    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd94d51-9423-41c3-8ab2-e94b52ab5904-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:42.801527 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:42.801464    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q7kv\" (UniqueName: \"kubernetes.io/projected/ecd94d51-9423-41c3-8ab2-e94b52ab5904-kube-api-access-7q7kv\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:43.516152 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:43.516123    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"
Oct 29 23:01:43.516352 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:43.516113    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj" event={"ID":"ecd94d51-9423-41c3-8ab2-e94b52ab5904","Type":"ContainerDied","Data":"67a08e97ff8ee8a6d2154a16cc214cfece5f2424f5eed69ef16cf75f05882618"}
Oct 29 23:01:43.516352 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:43.516281    4559 scope.go:122] "RemoveContainer" containerID="536623d37bcad3265f823d0d48ec46bb5096d871c44b055c3811335a8e0ac5fa"
Oct 29 23:01:43.572622 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:43.572582    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"]
Oct 29 23:01:43.584117 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:43.584087    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-08ebee-7bcfbb7487-9x5xj"]
Oct 29 23:01:43.704329 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:01:43.704292    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:01:45.229106 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:45.229070    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecd94d51-9423-41c3-8ab2-e94b52ab5904" path="/var/lib/kubelet/pods/ecd94d51-9423-41c3-8ab2-e94b52ab5904/volumes"
Oct 29 23:01:52.509618 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:52.509579    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"
Oct 29 23:01:52.510098 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:52.509669    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"
Oct 29 23:01:57.257402 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:57.257111    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"]
Oct 29 23:01:57.257402 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:57.257349    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887" podUID="6e878d3a-382f-4f23-bc40-e28385b96f5e" containerName="maas-api" containerID="cri-o://a7aa7fc3c5473378a03c955d83388869885d9f3bcf4d4ae315b0ecb3801f9f8b" gracePeriod=30
Oct 29 23:01:57.852234 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:57.851874    4559 generic.go:356] "Generic (PLEG): container finished" podID="6e878d3a-382f-4f23-bc40-e28385b96f5e" containerID="a7aa7fc3c5473378a03c955d83388869885d9f3bcf4d4ae315b0ecb3801f9f8b" exitCode=0
Oct 29 23:01:57.852234 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:57.852114    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887" event={"ID":"6e878d3a-382f-4f23-bc40-e28385b96f5e","Type":"ContainerDied","Data":"a7aa7fc3c5473378a03c955d83388869885d9f3bcf4d4ae315b0ecb3801f9f8b"}
Oct 29 23:01:57.884754 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:57.884724    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 23:01:57.916512 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:57.916479    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e878d3a-382f-4f23-bc40-e28385b96f5e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e878d3a-382f-4f23-bc40-e28385b96f5e-metrics-certs\") pod \"6e878d3a-382f-4f23-bc40-e28385b96f5e\" (UID: \"6e878d3a-382f-4f23-bc40-e28385b96f5e\") "
Oct 29 23:01:57.916703 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:57.916683    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e878d3a-382f-4f23-bc40-e28385b96f5e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e878d3a-382f-4f23-bc40-e28385b96f5e-maas-api-tls\") pod \"6e878d3a-382f-4f23-bc40-e28385b96f5e\" (UID: \"6e878d3a-382f-4f23-bc40-e28385b96f5e\") "
Oct 29 23:01:57.916746 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:57.916723    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e878d3a-382f-4f23-bc40-e28385b96f5e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e878d3a-382f-4f23-bc40-e28385b96f5e-openshift-service-ca\") pod \"6e878d3a-382f-4f23-bc40-e28385b96f5e\" (UID: \"6e878d3a-382f-4f23-bc40-e28385b96f5e\") "
Oct 29 23:01:57.916827 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:57.916791    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e878d3a-382f-4f23-bc40-e28385b96f5e-kube-api-access-f4z9p\" (UniqueName: \"kubernetes.io/projected/6e878d3a-382f-4f23-bc40-e28385b96f5e-kube-api-access-f4z9p\") pod \"6e878d3a-382f-4f23-bc40-e28385b96f5e\" (UID: \"6e878d3a-382f-4f23-bc40-e28385b96f5e\") "
Oct 29 23:01:57.917271 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:57.917195    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e878d3a-382f-4f23-bc40-e28385b96f5e-openshift-service-ca" pod "6e878d3a-382f-4f23-bc40-e28385b96f5e" (UID: "6e878d3a-382f-4f23-bc40-e28385b96f5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:57.917488 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:57.917445    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e878d3a-382f-4f23-bc40-e28385b96f5e-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:57.919147 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:57.919119    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e878d3a-382f-4f23-bc40-e28385b96f5e-maas-api-tls" pod "6e878d3a-382f-4f23-bc40-e28385b96f5e" (UID: "6e878d3a-382f-4f23-bc40-e28385b96f5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:57.919253 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:57.919146    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e878d3a-382f-4f23-bc40-e28385b96f5e-metrics-certs" pod "6e878d3a-382f-4f23-bc40-e28385b96f5e" (UID: "6e878d3a-382f-4f23-bc40-e28385b96f5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:57.919253 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:57.919151    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e878d3a-382f-4f23-bc40-e28385b96f5e-kube-api-access-f4z9p" pod "6e878d3a-382f-4f23-bc40-e28385b96f5e" (UID: "6e878d3a-382f-4f23-bc40-e28385b96f5e"). InnerVolumeSpecName "kube-api-access-f4z9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:57.974942 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:57.974899    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"]
Oct 29 23:01:57.981326 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:57.981280    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-43b37d-2m2ln"]
Oct 29 23:01:58.018583 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:58.018534    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e878d3a-382f-4f23-bc40-e28385b96f5e-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:58.018583 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:58.018577    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e878d3a-382f-4f23-bc40-e28385b96f5e-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:58.018827 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:58.018591    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4z9p\" (UniqueName: \"kubernetes.io/projected/6e878d3a-382f-4f23-bc40-e28385b96f5e-kube-api-access-f4z9p\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:01:58.863007 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:58.862973    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"
Oct 29 23:01:58.863544 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:58.863010    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887" event={"ID":"6e878d3a-382f-4f23-bc40-e28385b96f5e","Type":"ContainerDied","Data":"305f766acaa669cbb56dec126615089c16ff390cdc1445c54481aadcca1f48e4"}
Oct 29 23:01:58.863544 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:58.863082    4559 scope.go:122] "RemoveContainer" containerID="a7aa7fc3c5473378a03c955d83388869885d9f3bcf4d4ae315b0ecb3801f9f8b"
Oct 29 23:01:58.915188 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:58.915149    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"]
Oct 29 23:01:58.927923 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:58.927887    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887"]
Oct 29 23:01:59.228942 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:59.228908    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3" path="/var/lib/kubelet/pods/5f03ba23-9c3b-4e0d-bb48-c5efe44e4cf3/volumes"
Oct 29 23:01:59.229402 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:59.229387    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e878d3a-382f-4f23-bc40-e28385b96f5e" path="/var/lib/kubelet/pods/6e878d3a-382f-4f23-bc40-e28385b96f5e/volumes"
Oct 29 23:01:59.555440 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:59.555351    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"]
Oct 29 23:01:59.555622 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:59.555575    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7" podUID="732da306-43ba-4e56-a01f-70532d0211cb" containerName="payload-processing" containerID="cri-o://f7d3ced85cd99fc7c2f0f8666acab2f781945056d89a19aa941a766596a44d82" gracePeriod=30
Oct 29 23:01:59.850065 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:59.850034    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"
Oct 29 23:01:59.961191 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:59.960498    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/732da306-43ba-4e56-a01f-70532d0211cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732da306-43ba-4e56-a01f-70532d0211cb-plugins-config-volume\") pod \"732da306-43ba-4e56-a01f-70532d0211cb\" (UID: \"732da306-43ba-4e56-a01f-70532d0211cb\") "
Oct 29 23:01:59.962005 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:59.961978    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732da306-43ba-4e56-a01f-70532d0211cb-kube-api-access-kkbqp\" (UniqueName: \"kubernetes.io/projected/732da306-43ba-4e56-a01f-70532d0211cb-kube-api-access-kkbqp\") pod \"732da306-43ba-4e56-a01f-70532d0211cb\" (UID: \"732da306-43ba-4e56-a01f-70532d0211cb\") "
Oct 29 23:01:59.963169 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:59.961151    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732da306-43ba-4e56-a01f-70532d0211cb-plugins-config-volume" pod "732da306-43ba-4e56-a01f-70532d0211cb" (UID: "732da306-43ba-4e56-a01f-70532d0211cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:00.007142 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:59.992094    4559 generic.go:356] "Generic (PLEG): container finished" podID="732da306-43ba-4e56-a01f-70532d0211cb" containerID="f7d3ced85cd99fc7c2f0f8666acab2f781945056d89a19aa941a766596a44d82" exitCode=0
Oct 29 23:02:00.007142 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:59.992219    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7" event={"ID":"732da306-43ba-4e56-a01f-70532d0211cb","Type":"ContainerDied","Data":"f7d3ced85cd99fc7c2f0f8666acab2f781945056d89a19aa941a766596a44d82"}
Oct 29 23:02:00.007142 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:59.992331    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7" event={"ID":"732da306-43ba-4e56-a01f-70532d0211cb","Type":"ContainerDied","Data":"5566170a496914a39167e47fe743bab4841da1a1f0ded7336e23c0b628d0534c"}
Oct 29 23:02:00.007142 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:59.992412    4559 scope.go:122] "RemoveContainer" containerID="f7d3ced85cd99fc7c2f0f8666acab2f781945056d89a19aa941a766596a44d82"
Oct 29 23:02:00.007142 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:01:59.992580    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"
Oct 29 23:02:00.064936 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:00.064883    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732da306-43ba-4e56-a01f-70532d0211cb-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:00.108073 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:00.106152    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732da306-43ba-4e56-a01f-70532d0211cb-kube-api-access-kkbqp" pod "732da306-43ba-4e56-a01f-70532d0211cb" (UID: "732da306-43ba-4e56-a01f-70532d0211cb"). InnerVolumeSpecName "kube-api-access-kkbqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:00.112560 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:00.112528    4559 scope.go:122] "RemoveContainer" containerID="f7d3ced85cd99fc7c2f0f8666acab2f781945056d89a19aa941a766596a44d82"
Oct 29 23:02:00.113509 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:02:00.113079    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d3ced85cd99fc7c2f0f8666acab2f781945056d89a19aa941a766596a44d82\": container with ID starting with f7d3ced85cd99fc7c2f0f8666acab2f781945056d89a19aa941a766596a44d82 not found: ID does not exist" containerID="f7d3ced85cd99fc7c2f0f8666acab2f781945056d89a19aa941a766596a44d82"
Oct 29 23:02:00.113509 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:00.113122    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d3ced85cd99fc7c2f0f8666acab2f781945056d89a19aa941a766596a44d82"} err="failed to get container status \"f7d3ced85cd99fc7c2f0f8666acab2f781945056d89a19aa941a766596a44d82\": rpc error: code = NotFound desc = could not find container \"f7d3ced85cd99fc7c2f0f8666acab2f781945056d89a19aa941a766596a44d82\": container with ID starting with f7d3ced85cd99fc7c2f0f8666acab2f781945056d89a19aa941a766596a44d82 not found: ID does not exist"
Oct 29 23:02:00.124562 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:00.124526    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"]
Oct 29 23:02:00.128340 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:00.128305    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75b68d-zlbfv"]
Oct 29 23:02:00.166546 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:00.166503    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkbqp\" (UniqueName: \"kubernetes.io/projected/732da306-43ba-4e56-a01f-70532d0211cb-kube-api-access-kkbqp\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:00.340772 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:00.340724    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"]
Oct 29 23:02:00.345071 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:00.345034    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75b68d-ddbc7885b-wmsp7"]
Oct 29 23:02:00.728360 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:00.728320    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-43b37d-c56588694-v4887" podUID="6e878d3a-382f-4f23-bc40-e28385b96f5e" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:02:01.229162 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:01.229129    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732da306-43ba-4e56-a01f-70532d0211cb" path="/var/lib/kubelet/pods/732da306-43ba-4e56-a01f-70532d0211cb/volumes"
Oct 29 23:02:01.229607 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:01.229591    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f3b306-0b68-4152-98c0-960491150159" path="/var/lib/kubelet/pods/c8f3b306-0b68-4152-98c0-960491150159/volumes"
Oct 29 23:02:02.510158 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:02.510124    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:02:03.865289 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:03.865252    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"]
Oct 29 23:02:03.865994 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:03.865979    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ecd94d51-9423-41c3-8ab2-e94b52ab5904" containerName="payload-pre-processing"
Oct 29 23:02:03.866072 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:03.865994    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd94d51-9423-41c3-8ab2-e94b52ab5904" containerName="payload-pre-processing"
Oct 29 23:02:03.866072 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:03.866035    4559 cpu_manager.go:414] "containerMap: removing container" podUID="732da306-43ba-4e56-a01f-70532d0211cb" containerName="payload-processing"
Oct 29 23:02:03.866072 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:03.866040    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732da306-43ba-4e56-a01f-70532d0211cb" containerName="payload-processing"
Oct 29 23:02:03.866072 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:03.866062    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6e878d3a-382f-4f23-bc40-e28385b96f5e" containerName="maas-api"
Oct 29 23:02:03.866072 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:03.866066    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e878d3a-382f-4f23-bc40-e28385b96f5e" containerName="maas-api"
Oct 29 23:02:03.866248 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:03.866154    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="732da306-43ba-4e56-a01f-70532d0211cb" containerName="payload-processing"
Oct 29 23:02:03.866248 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:03.866167    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e878d3a-382f-4f23-bc40-e28385b96f5e" containerName="maas-api"
Oct 29 23:02:03.866248 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:03.866175    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd94d51-9423-41c3-8ab2-e94b52ab5904" containerName="payload-pre-processing"
Oct 29 23:02:03.871357 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:03.871332    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:02:03.891856 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:03.891776    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"]
Oct 29 23:02:03.894991 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:03.894963    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9d6d4e-serving-cert\""
Oct 29 23:02:03.967035 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:03.966998    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1884171-dc00-4a39-8181-75468d0ba179-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5\" (UID: \"c1884171-dc00-4a39-8181-75468d0ba179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:02:03.967266 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:03.967049    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1884171-dc00-4a39-8181-75468d0ba179-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5\" (UID: \"c1884171-dc00-4a39-8181-75468d0ba179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:02:03.967266 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:03.967221    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1884171-dc00-4a39-8181-75468d0ba179-metrics-certs\") pod \"maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5\" (UID: \"c1884171-dc00-4a39-8181-75468d0ba179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:02:03.967393 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:03.967263    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k88mr\" (UniqueName: \"kubernetes.io/projected/c1884171-dc00-4a39-8181-75468d0ba179-kube-api-access-k88mr\") pod \"maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5\" (UID: \"c1884171-dc00-4a39-8181-75468d0ba179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:02:04.087836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:04.083827    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1884171-dc00-4a39-8181-75468d0ba179-metrics-certs\") pod \"maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5\" (UID: \"c1884171-dc00-4a39-8181-75468d0ba179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:02:04.087836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:04.083901    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k88mr\" (UniqueName: \"kubernetes.io/projected/c1884171-dc00-4a39-8181-75468d0ba179-kube-api-access-k88mr\") pod \"maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5\" (UID: \"c1884171-dc00-4a39-8181-75468d0ba179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:02:04.087836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:04.084108    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1884171-dc00-4a39-8181-75468d0ba179-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5\" (UID: \"c1884171-dc00-4a39-8181-75468d0ba179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:02:04.087836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:04.084157    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1884171-dc00-4a39-8181-75468d0ba179-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5\" (UID: \"c1884171-dc00-4a39-8181-75468d0ba179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:02:04.087836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:04.087399    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1884171-dc00-4a39-8181-75468d0ba179-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5\" (UID: \"c1884171-dc00-4a39-8181-75468d0ba179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:02:04.104833 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:04.104774    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1884171-dc00-4a39-8181-75468d0ba179-metrics-certs\") pod \"maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5\" (UID: \"c1884171-dc00-4a39-8181-75468d0ba179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:02:04.110604 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:04.110475    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1884171-dc00-4a39-8181-75468d0ba179-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5\" (UID: \"c1884171-dc00-4a39-8181-75468d0ba179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:02:04.113610 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:04.113580    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k88mr\" (UniqueName: \"kubernetes.io/projected/c1884171-dc00-4a39-8181-75468d0ba179-kube-api-access-k88mr\") pod \"maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5\" (UID: \"c1884171-dc00-4a39-8181-75468d0ba179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:02:04.192192 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:04.192154    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:02:04.501307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:04.501281    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"]
Oct 29 23:02:04.502198 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:02:04.502166    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1884171_dc00_4a39_8181_75468d0ba179.slice/crio-bda9161b697ca5f74bd51b546dc24bf8599212d5abc0bac2fbca4bd60f5a46a0 WatchSource:0}: Error finding container bda9161b697ca5f74bd51b546dc24bf8599212d5abc0bac2fbca4bd60f5a46a0: Status 404 returned error can't find the container with id bda9161b697ca5f74bd51b546dc24bf8599212d5abc0bac2fbca4bd60f5a46a0
Oct 29 23:02:04.504147 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:04.504129    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:02:05.039705 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:05.039661    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5" event={"ID":"c1884171-dc00-4a39-8181-75468d0ba179","Type":"ContainerStarted","Data":"fa2a38eaff5b40cb4a2b9fc17ac80bfa9990079e809a1290904c55e3b1d07414"}
Oct 29 23:02:05.040246 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:05.039776    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5" event={"ID":"c1884171-dc00-4a39-8181-75468d0ba179","Type":"ContainerStarted","Data":"bda9161b697ca5f74bd51b546dc24bf8599212d5abc0bac2fbca4bd60f5a46a0"}
Oct 29 23:02:05.040246 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:05.039930    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:02:05.108127 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:05.108060    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5" podStartSLOduration=1.738284607 podStartE2EDuration="2.108038934s" totalImagesPullingTime="369.754327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:05.092053128 +0000 UTC m=+2864.003496682" watchObservedRunningTime="2026-10-29 23:02:05.108038934 +0000 UTC m=+2864.019482489"
Oct 29 23:02:05.112384 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:05.112350    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"]
Oct 29 23:02:05.112635 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:05.112598    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67" podUID="52d2294b-e879-4782-b9e9-526f2aeecdba" containerName="istio-proxy" containerID="cri-o://ad591b2804aa639333022edfdeae1fbfb39cb05cb1935954c11abd6a61cd64e0" gracePeriod=30
Oct 29 23:02:05.421945 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:05.421901    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"]
Oct 29 23:02:05.422922 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:05.422885    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5" podUID="bb1155cc-6ac9-44d6-bd62-e09f88841686" containerName="maas-api" containerID="cri-o://49752ad01da7019136dfe66e8acd03b398b13a84c9fd24bff33a3facf7983730" gracePeriod=30
Oct 29 23:02:05.849870 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:05.847882    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 23:02:05.944833 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:05.942301    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1155cc-6ac9-44d6-bd62-e09f88841686-kube-api-access-hbtll\" (UniqueName: \"kubernetes.io/projected/bb1155cc-6ac9-44d6-bd62-e09f88841686-kube-api-access-hbtll\") pod \"bb1155cc-6ac9-44d6-bd62-e09f88841686\" (UID: \"bb1155cc-6ac9-44d6-bd62-e09f88841686\") "
Oct 29 23:02:05.944833 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:05.942484    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb1155cc-6ac9-44d6-bd62-e09f88841686-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb1155cc-6ac9-44d6-bd62-e09f88841686-metrics-certs\") pod \"bb1155cc-6ac9-44d6-bd62-e09f88841686\" (UID: \"bb1155cc-6ac9-44d6-bd62-e09f88841686\") "
Oct 29 23:02:05.944833 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:05.942520    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb1155cc-6ac9-44d6-bd62-e09f88841686-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb1155cc-6ac9-44d6-bd62-e09f88841686-maas-api-tls\") pod \"bb1155cc-6ac9-44d6-bd62-e09f88841686\" (UID: \"bb1155cc-6ac9-44d6-bd62-e09f88841686\") "
Oct 29 23:02:05.944833 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:05.942637    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb1155cc-6ac9-44d6-bd62-e09f88841686-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb1155cc-6ac9-44d6-bd62-e09f88841686-openshift-service-ca\") pod \"bb1155cc-6ac9-44d6-bd62-e09f88841686\" (UID: \"bb1155cc-6ac9-44d6-bd62-e09f88841686\") "
Oct 29 23:02:05.972857 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:05.950037    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb1155cc-6ac9-44d6-bd62-e09f88841686-openshift-service-ca" pod "bb1155cc-6ac9-44d6-bd62-e09f88841686" (UID: "bb1155cc-6ac9-44d6-bd62-e09f88841686"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:05.994327 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:05.988142    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb1155cc-6ac9-44d6-bd62-e09f88841686-maas-api-tls" pod "bb1155cc-6ac9-44d6-bd62-e09f88841686" (UID: "bb1155cc-6ac9-44d6-bd62-e09f88841686"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:06.036297 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:06.036262    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1155cc-6ac9-44d6-bd62-e09f88841686-kube-api-access-hbtll" pod "bb1155cc-6ac9-44d6-bd62-e09f88841686" (UID: "bb1155cc-6ac9-44d6-bd62-e09f88841686"). InnerVolumeSpecName "kube-api-access-hbtll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:06.055576 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:06.054078    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbtll\" (UniqueName: \"kubernetes.io/projected/bb1155cc-6ac9-44d6-bd62-e09f88841686-kube-api-access-hbtll\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:06.055576 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:06.054114    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb1155cc-6ac9-44d6-bd62-e09f88841686-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:06.055576 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:06.054129    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb1155cc-6ac9-44d6-bd62-e09f88841686-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:06.059361 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:06.059307    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb1155cc-6ac9-44d6-bd62-e09f88841686-metrics-certs" pod "bb1155cc-6ac9-44d6-bd62-e09f88841686" (UID: "bb1155cc-6ac9-44d6-bd62-e09f88841686"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:06.067775 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:06.067662    4559 generic.go:356] "Generic (PLEG): container finished" podID="bb1155cc-6ac9-44d6-bd62-e09f88841686" containerID="49752ad01da7019136dfe66e8acd03b398b13a84c9fd24bff33a3facf7983730" exitCode=0
Oct 29 23:02:06.068836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:06.068402    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5" event={"ID":"bb1155cc-6ac9-44d6-bd62-e09f88841686","Type":"ContainerDied","Data":"49752ad01da7019136dfe66e8acd03b398b13a84c9fd24bff33a3facf7983730"}
Oct 29 23:02:06.068836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:06.068432    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"
Oct 29 23:02:06.068836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:06.068458    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5" event={"ID":"bb1155cc-6ac9-44d6-bd62-e09f88841686","Type":"ContainerDied","Data":"b468aa38423ea132cbbd2c6c98b3ba66b5dc4c64930b2953440562c90e21a061"}
Oct 29 23:02:06.068836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:06.068488    4559 scope.go:122] "RemoveContainer" containerID="49752ad01da7019136dfe66e8acd03b398b13a84c9fd24bff33a3facf7983730"
Oct 29 23:02:06.082181 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:06.082143    4559 scope.go:122] "RemoveContainer" containerID="49752ad01da7019136dfe66e8acd03b398b13a84c9fd24bff33a3facf7983730"
Oct 29 23:02:06.082564 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:02:06.082541    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49752ad01da7019136dfe66e8acd03b398b13a84c9fd24bff33a3facf7983730\": container with ID starting with 49752ad01da7019136dfe66e8acd03b398b13a84c9fd24bff33a3facf7983730 not found: ID does not exist" containerID="49752ad01da7019136dfe66e8acd03b398b13a84c9fd24bff33a3facf7983730"
Oct 29 23:02:06.082642 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:06.082575    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49752ad01da7019136dfe66e8acd03b398b13a84c9fd24bff33a3facf7983730"} err="failed to get container status \"49752ad01da7019136dfe66e8acd03b398b13a84c9fd24bff33a3facf7983730\": rpc error: code = NotFound desc = could not find container \"49752ad01da7019136dfe66e8acd03b398b13a84c9fd24bff33a3facf7983730\": container with ID starting with 49752ad01da7019136dfe66e8acd03b398b13a84c9fd24bff33a3facf7983730 not found: ID does not exist"
Oct 29 23:02:06.119244 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:06.119214    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"]
Oct 29 23:02:06.125148 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:06.125115    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-08ebee-9qwxv"]
Oct 29 23:02:06.130895 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:06.130864    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"]
Oct 29 23:02:06.136302 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:06.136275    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-08ebee-9d5444fd5-fxxw5"]
Oct 29 23:02:06.183835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:06.166056    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb1155cc-6ac9-44d6-bd62-e09f88841686-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:07.229022 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:07.228983    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb1155cc-6ac9-44d6-bd62-e09f88841686" path="/var/lib/kubelet/pods/bb1155cc-6ac9-44d6-bd62-e09f88841686/volumes"
Oct 29 23:02:07.229495 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:07.229479    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de7c3068-21b6-4ae0-9f35-c60e986538c7" path="/var/lib/kubelet/pods/de7c3068-21b6-4ae0-9f35-c60e986538c7/volumes"
Oct 29 23:02:07.527066 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:07.526979    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"]
Oct 29 23:02:07.527251 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:07.527186    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn" podUID="6f118086-bc70-4d6d-874a-12ebec737d0d" containerName="istio-proxy" containerID="cri-o://59c509d1c569f337f550c077bbe52831b0e237ddf1f21a3c5a355da8444f2732" gracePeriod=30
Oct 29 23:02:08.019534 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.019493    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"]
Oct 29 23:02:08.020340 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.020316    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bb1155cc-6ac9-44d6-bd62-e09f88841686" containerName="maas-api"
Oct 29 23:02:08.020340 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.020339    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb1155cc-6ac9-44d6-bd62-e09f88841686" containerName="maas-api"
Oct 29 23:02:08.020553 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.020453    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb1155cc-6ac9-44d6-bd62-e09f88841686" containerName="maas-api"
Oct 29 23:02:08.025462 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.025429    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"
Oct 29 23:02:08.041365 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.041327    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:02:08.041550 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.041375    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:02:08.053781 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.053745    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"]
Oct 29 23:02:08.133969 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.133928    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbtm8\" (UniqueName: \"kubernetes.io/projected/5022bff6-abb7-4762-87b0-aa1b6f71ab70-kube-api-access-xbtm8\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7\" (UID: \"5022bff6-abb7-4762-87b0-aa1b6f71ab70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"
Oct 29 23:02:08.138020 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.134366    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5022bff6-abb7-4762-87b0-aa1b6f71ab70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7\" (UID: \"5022bff6-abb7-4762-87b0-aa1b6f71ab70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"
Oct 29 23:02:08.138020 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.134494    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5022bff6-abb7-4762-87b0-aa1b6f71ab70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7\" (UID: \"5022bff6-abb7-4762-87b0-aa1b6f71ab70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"
Oct 29 23:02:08.236167 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.236125    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbtm8\" (UniqueName: \"kubernetes.io/projected/5022bff6-abb7-4762-87b0-aa1b6f71ab70-kube-api-access-xbtm8\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7\" (UID: \"5022bff6-abb7-4762-87b0-aa1b6f71ab70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"
Oct 29 23:02:08.236632 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.236285    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5022bff6-abb7-4762-87b0-aa1b6f71ab70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7\" (UID: \"5022bff6-abb7-4762-87b0-aa1b6f71ab70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"
Oct 29 23:02:08.236632 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.236341    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5022bff6-abb7-4762-87b0-aa1b6f71ab70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7\" (UID: \"5022bff6-abb7-4762-87b0-aa1b6f71ab70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"
Oct 29 23:02:08.237066 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.237036    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5022bff6-abb7-4762-87b0-aa1b6f71ab70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7\" (UID: \"5022bff6-abb7-4762-87b0-aa1b6f71ab70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"
Oct 29 23:02:08.238892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.238867    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5022bff6-abb7-4762-87b0-aa1b6f71ab70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7\" (UID: \"5022bff6-abb7-4762-87b0-aa1b6f71ab70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"
Oct 29 23:02:08.256173 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.256145    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbtm8\" (UniqueName: \"kubernetes.io/projected/5022bff6-abb7-4762-87b0-aa1b6f71ab70-kube-api-access-xbtm8\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7\" (UID: \"5022bff6-abb7-4762-87b0-aa1b6f71ab70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"
Oct 29 23:02:08.346988 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.346898    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"
Oct 29 23:02:08.650856 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:08.650822    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"]
Oct 29 23:02:08.651657 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:02:08.651627    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5022bff6_abb7_4762_87b0_aa1b6f71ab70.slice/crio-a241643a49091cbd14d692019308a4b2c85f03718bef5511d155eff7eeee898e WatchSource:0}: Error finding container a241643a49091cbd14d692019308a4b2c85f03718bef5511d155eff7eeee898e: Status 404 returned error can't find the container with id a241643a49091cbd14d692019308a4b2c85f03718bef5511d155eff7eeee898e
Oct 29 23:02:09.167489 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:09.167442    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7" event={"ID":"5022bff6-abb7-4762-87b0-aa1b6f71ab70","Type":"ContainerStarted","Data":"a8153c76316ac00a2e6385e72e308c43037baa0b388e94898e365ae83c183d09"}
Oct 29 23:02:09.167722 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:09.167527    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7" event={"ID":"5022bff6-abb7-4762-87b0-aa1b6f71ab70","Type":"ContainerStarted","Data":"a241643a49091cbd14d692019308a4b2c85f03718bef5511d155eff7eeee898e"}
Oct 29 23:02:09.207170 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:09.207119    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7" podStartSLOduration=2.207103173 podStartE2EDuration="2.207103173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:09.206361903 +0000 UTC m=+2868.117805458" watchObservedRunningTime="2026-10-29 23:02:09.207103173 +0000 UTC m=+2868.118546724"
Oct 29 23:02:10.177201 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.177162    4559 generic.go:356] "Generic (PLEG): container finished" podID="5022bff6-abb7-4762-87b0-aa1b6f71ab70" containerID="a8153c76316ac00a2e6385e72e308c43037baa0b388e94898e365ae83c183d09" exitCode=0
Oct 29 23:02:10.177633 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.177240    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7" event={"ID":"5022bff6-abb7-4762-87b0-aa1b6f71ab70","Type":"ContainerDied","Data":"a8153c76316ac00a2e6385e72e308c43037baa0b388e94898e365ae83c183d09"}
Oct 29 23:02:10.422054 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.422022    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 23:02:10.474615 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.474575    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-podinfo\") pod \"52d2294b-e879-4782-b9e9-526f2aeecdba\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") "
Oct 29 23:02:10.475236 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.475201    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-envoy\") pod \"52d2294b-e879-4782-b9e9-526f2aeecdba\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") "
Oct 29 23:02:10.475447 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.475296    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52d2294b-e879-4782-b9e9-526f2aeecdba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d2294b-e879-4782-b9e9-526f2aeecdba-istiod-ca-cert\") pod \"52d2294b-e879-4782-b9e9-526f2aeecdba\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") "
Oct 29 23:02:10.475447 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.475334    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-workload-certs\") pod \"52d2294b-e879-4782-b9e9-526f2aeecdba\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") "
Oct 29 23:02:10.475447 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.475434    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-data\") pod \"52d2294b-e879-4782-b9e9-526f2aeecdba\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") "
Oct 29 23:02:10.475689 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.475477    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d2294b-e879-4782-b9e9-526f2aeecdba-kube-api-access-88mzx\" (UniqueName: \"kubernetes.io/projected/52d2294b-e879-4782-b9e9-526f2aeecdba-kube-api-access-88mzx\") pod \"52d2294b-e879-4782-b9e9-526f2aeecdba\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") "
Oct 29 23:02:10.476600 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.476568    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-token\" (UniqueName: \"kubernetes.io/projected/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-token\") pod \"52d2294b-e879-4782-b9e9-526f2aeecdba\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") "
Oct 29 23:02:10.476737 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.476631    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-credential-socket\") pod \"52d2294b-e879-4782-b9e9-526f2aeecdba\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") "
Oct 29 23:02:10.476885 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.476859    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-workload-socket\") pod \"52d2294b-e879-4782-b9e9-526f2aeecdba\" (UID: \"52d2294b-e879-4782-b9e9-526f2aeecdba\") "
Oct 29 23:02:10.477880 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.477849    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-workload-socket" pod "52d2294b-e879-4782-b9e9-526f2aeecdba" (UID: "52d2294b-e879-4782-b9e9-526f2aeecdba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:10.479306 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.479276    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-credential-socket" pod "52d2294b-e879-4782-b9e9-526f2aeecdba" (UID: "52d2294b-e879-4782-b9e9-526f2aeecdba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:10.479636 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.479609    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-workload-certs" pod "52d2294b-e879-4782-b9e9-526f2aeecdba" (UID: "52d2294b-e879-4782-b9e9-526f2aeecdba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:10.480378 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.480349    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-data" pod "52d2294b-e879-4782-b9e9-526f2aeecdba" (UID: "52d2294b-e879-4782-b9e9-526f2aeecdba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:10.482923 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.482638    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-podinfo" pod "52d2294b-e879-4782-b9e9-526f2aeecdba" (UID: "52d2294b-e879-4782-b9e9-526f2aeecdba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:10.483655 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.483627    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d2294b-e879-4782-b9e9-526f2aeecdba-istiod-ca-cert" pod "52d2294b-e879-4782-b9e9-526f2aeecdba" (UID: "52d2294b-e879-4782-b9e9-526f2aeecdba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:10.484279 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.484240    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d2294b-e879-4782-b9e9-526f2aeecdba-kube-api-access-88mzx" pod "52d2294b-e879-4782-b9e9-526f2aeecdba" (UID: "52d2294b-e879-4782-b9e9-526f2aeecdba"). InnerVolumeSpecName "kube-api-access-88mzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:10.484682 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.484639    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-envoy" pod "52d2294b-e879-4782-b9e9-526f2aeecdba" (UID: "52d2294b-e879-4782-b9e9-526f2aeecdba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:10.485353 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.485321    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-token" pod "52d2294b-e879-4782-b9e9-526f2aeecdba" (UID: "52d2294b-e879-4782-b9e9-526f2aeecdba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:10.579430 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.579385    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.579430 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.579428    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.579662 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.579442    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.579662 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.579454    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d2294b-e879-4782-b9e9-526f2aeecdba-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.579662 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.579466    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.579662 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.579475    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.579662 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.579488    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88mzx\" (UniqueName: \"kubernetes.io/projected/52d2294b-e879-4782-b9e9-526f2aeecdba-kube-api-access-88mzx\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.579662 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.579499    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d2294b-e879-4782-b9e9-526f2aeecdba-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:10.579662 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:10.579510    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d2294b-e879-4782-b9e9-526f2aeecdba-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:11.077674 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.077640    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:02:11.189875 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.189835    4559 generic.go:356] "Generic (PLEG): container finished" podID="52d2294b-e879-4782-b9e9-526f2aeecdba" containerID="ad591b2804aa639333022edfdeae1fbfb39cb05cb1935954c11abd6a61cd64e0" exitCode=0
Oct 29 23:02:11.190343 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.189900    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"
Oct 29 23:02:11.190343 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.189917    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67" event={"ID":"52d2294b-e879-4782-b9e9-526f2aeecdba","Type":"ContainerDied","Data":"ad591b2804aa639333022edfdeae1fbfb39cb05cb1935954c11abd6a61cd64e0"}
Oct 29 23:02:11.190343 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.189982    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67" event={"ID":"52d2294b-e879-4782-b9e9-526f2aeecdba","Type":"ContainerDied","Data":"658098d36f116eb6db0a4ded0d37c2680e2fbbb9dfcfe1728bcae3e92758e428"}
Oct 29 23:02:11.190343 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.190013    4559 scope.go:122] "RemoveContainer" containerID="ad591b2804aa639333022edfdeae1fbfb39cb05cb1935954c11abd6a61cd64e0"
Oct 29 23:02:11.207763 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.207627    4559 scope.go:122] "RemoveContainer" containerID="db9a82df3cf9c2a7fe143ba9bccb4fd7357a9d9dca1c66fe3b8636837fa1adc7"
Oct 29 23:02:11.222429 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.222401    4559 scope.go:122] "RemoveContainer" containerID="ad591b2804aa639333022edfdeae1fbfb39cb05cb1935954c11abd6a61cd64e0"
Oct 29 23:02:11.222824 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:02:11.222782    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad591b2804aa639333022edfdeae1fbfb39cb05cb1935954c11abd6a61cd64e0\": container with ID starting with ad591b2804aa639333022edfdeae1fbfb39cb05cb1935954c11abd6a61cd64e0 not found: ID does not exist" containerID="ad591b2804aa639333022edfdeae1fbfb39cb05cb1935954c11abd6a61cd64e0"
Oct 29 23:02:11.222906 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.222848    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad591b2804aa639333022edfdeae1fbfb39cb05cb1935954c11abd6a61cd64e0"} err="failed to get container status \"ad591b2804aa639333022edfdeae1fbfb39cb05cb1935954c11abd6a61cd64e0\": rpc error: code = NotFound desc = could not find container \"ad591b2804aa639333022edfdeae1fbfb39cb05cb1935954c11abd6a61cd64e0\": container with ID starting with ad591b2804aa639333022edfdeae1fbfb39cb05cb1935954c11abd6a61cd64e0 not found: ID does not exist"
Oct 29 23:02:11.222906 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.222874    4559 scope.go:122] "RemoveContainer" containerID="db9a82df3cf9c2a7fe143ba9bccb4fd7357a9d9dca1c66fe3b8636837fa1adc7"
Oct 29 23:02:11.223227 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:02:11.223204    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db9a82df3cf9c2a7fe143ba9bccb4fd7357a9d9dca1c66fe3b8636837fa1adc7\": container with ID starting with db9a82df3cf9c2a7fe143ba9bccb4fd7357a9d9dca1c66fe3b8636837fa1adc7 not found: ID does not exist" containerID="db9a82df3cf9c2a7fe143ba9bccb4fd7357a9d9dca1c66fe3b8636837fa1adc7"
Oct 29 23:02:11.223301 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.223237    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db9a82df3cf9c2a7fe143ba9bccb4fd7357a9d9dca1c66fe3b8636837fa1adc7"} err="failed to get container status \"db9a82df3cf9c2a7fe143ba9bccb4fd7357a9d9dca1c66fe3b8636837fa1adc7\": rpc error: code = NotFound desc = could not find container \"db9a82df3cf9c2a7fe143ba9bccb4fd7357a9d9dca1c66fe3b8636837fa1adc7\": container with ID starting with db9a82df3cf9c2a7fe143ba9bccb4fd7357a9d9dca1c66fe3b8636837fa1adc7 not found: ID does not exist"
Oct 29 23:02:11.232627 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.231618    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"]
Oct 29 23:02:11.232627 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.232208    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv" podUID="dbc13347-02e3-42dd-88f6-bf1503c428ef" containerName="payload-processing" containerID="cri-o://8661bc9a6c87fb9f5a5182d59543a3185c3c2cff22eff6d0fbf649f4996b5512" gracePeriod=30
Oct 29 23:02:11.240210 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.240183    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"]
Oct 29 23:02:11.240615 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.240588    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr" podUID="28022dfc-ac33-46b0-ba0e-066502f8acd5" containerName="payload-pre-processing" containerID="cri-o://6868a61a9a84f616291d93c3379c91f0dd118c4ffd82bc5b1feaa8984002521f" gracePeriod=30
Oct 29 23:02:11.254529 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.254497    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"]
Oct 29 23:02:11.284081 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.283867    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-cfx67"]
Oct 29 23:02:11.500561 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.500526    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"
Oct 29 23:02:11.624029 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.612394    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5022bff6-abb7-4762-87b0-aa1b6f71ab70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5022bff6-abb7-4762-87b0-aa1b6f71ab70-maas-api-openshift-ca\") pod \"5022bff6-abb7-4762-87b0-aa1b6f71ab70\" (UID: \"5022bff6-abb7-4762-87b0-aa1b6f71ab70\") "
Oct 29 23:02:11.624029 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.612460    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5022bff6-abb7-4762-87b0-aa1b6f71ab70-kube-api-access-xbtm8\" (UniqueName: \"kubernetes.io/projected/5022bff6-abb7-4762-87b0-aa1b6f71ab70-kube-api-access-xbtm8\") pod \"5022bff6-abb7-4762-87b0-aa1b6f71ab70\" (UID: \"5022bff6-abb7-4762-87b0-aa1b6f71ab70\") "
Oct 29 23:02:11.624029 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.612649    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5022bff6-abb7-4762-87b0-aa1b6f71ab70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5022bff6-abb7-4762-87b0-aa1b6f71ab70-maas-api-xks-ca\") pod \"5022bff6-abb7-4762-87b0-aa1b6f71ab70\" (UID: \"5022bff6-abb7-4762-87b0-aa1b6f71ab70\") "
Oct 29 23:02:11.628232 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.628189    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5022bff6-abb7-4762-87b0-aa1b6f71ab70-maas-api-openshift-ca" pod "5022bff6-abb7-4762-87b0-aa1b6f71ab70" (UID: "5022bff6-abb7-4762-87b0-aa1b6f71ab70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:11.698438 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.691056    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5022bff6-abb7-4762-87b0-aa1b6f71ab70-maas-api-xks-ca" pod "5022bff6-abb7-4762-87b0-aa1b6f71ab70" (UID: "5022bff6-abb7-4762-87b0-aa1b6f71ab70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:11.721168 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.721080    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5022bff6-abb7-4762-87b0-aa1b6f71ab70-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:11.721168 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.721125    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5022bff6-abb7-4762-87b0-aa1b6f71ab70-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:11.737008 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.736941    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5022bff6-abb7-4762-87b0-aa1b6f71ab70-kube-api-access-xbtm8" pod "5022bff6-abb7-4762-87b0-aa1b6f71ab70" (UID: "5022bff6-abb7-4762-87b0-aa1b6f71ab70"). InnerVolumeSpecName "kube-api-access-xbtm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:11.787124 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.786741    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"
Oct 29 23:02:11.823270 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.822505    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbc13347-02e3-42dd-88f6-bf1503c428ef-kube-api-access-g5ms8\" (UniqueName: \"kubernetes.io/projected/dbc13347-02e3-42dd-88f6-bf1503c428ef-kube-api-access-g5ms8\") pod \"dbc13347-02e3-42dd-88f6-bf1503c428ef\" (UID: \"dbc13347-02e3-42dd-88f6-bf1503c428ef\") "
Oct 29 23:02:11.823270 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.822769    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbc13347-02e3-42dd-88f6-bf1503c428ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbc13347-02e3-42dd-88f6-bf1503c428ef-plugins-config-volume\") pod \"dbc13347-02e3-42dd-88f6-bf1503c428ef\" (UID: \"dbc13347-02e3-42dd-88f6-bf1503c428ef\") "
Oct 29 23:02:11.823767 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.823692    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbtm8\" (UniqueName: \"kubernetes.io/projected/5022bff6-abb7-4762-87b0-aa1b6f71ab70-kube-api-access-xbtm8\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:11.824019 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.823992    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbc13347-02e3-42dd-88f6-bf1503c428ef-plugins-config-volume" pod "dbc13347-02e3-42dd-88f6-bf1503c428ef" (UID: "dbc13347-02e3-42dd-88f6-bf1503c428ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:11.825697 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.825673    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc13347-02e3-42dd-88f6-bf1503c428ef-kube-api-access-g5ms8" pod "dbc13347-02e3-42dd-88f6-bf1503c428ef" (UID: "dbc13347-02e3-42dd-88f6-bf1503c428ef"). InnerVolumeSpecName "kube-api-access-g5ms8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:11.898868 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.898419    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"
Oct 29 23:02:11.925435 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.925379    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5ms8\" (UniqueName: \"kubernetes.io/projected/dbc13347-02e3-42dd-88f6-bf1503c428ef-kube-api-access-g5ms8\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:11.925435 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:11.925435    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbc13347-02e3-42dd-88f6-bf1503c428ef-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:12.027146 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.027110    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28022dfc-ac33-46b0-ba0e-066502f8acd5-kube-api-access-l695m\" (UniqueName: \"kubernetes.io/projected/28022dfc-ac33-46b0-ba0e-066502f8acd5-kube-api-access-l695m\") pod \"28022dfc-ac33-46b0-ba0e-066502f8acd5\" (UID: \"28022dfc-ac33-46b0-ba0e-066502f8acd5\") "
Oct 29 23:02:12.027378 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.027192    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28022dfc-ac33-46b0-ba0e-066502f8acd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28022dfc-ac33-46b0-ba0e-066502f8acd5-plugins-config-volume\") pod \"28022dfc-ac33-46b0-ba0e-066502f8acd5\" (UID: \"28022dfc-ac33-46b0-ba0e-066502f8acd5\") "
Oct 29 23:02:12.027646 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.027617    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28022dfc-ac33-46b0-ba0e-066502f8acd5-plugins-config-volume" pod "28022dfc-ac33-46b0-ba0e-066502f8acd5" (UID: "28022dfc-ac33-46b0-ba0e-066502f8acd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:12.027947 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.027926    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28022dfc-ac33-46b0-ba0e-066502f8acd5-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:12.029999 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.029969    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28022dfc-ac33-46b0-ba0e-066502f8acd5-kube-api-access-l695m" pod "28022dfc-ac33-46b0-ba0e-066502f8acd5" (UID: "28022dfc-ac33-46b0-ba0e-066502f8acd5"). InnerVolumeSpecName "kube-api-access-l695m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:12.129610 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.129569    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l695m\" (UniqueName: \"kubernetes.io/projected/28022dfc-ac33-46b0-ba0e-066502f8acd5-kube-api-access-l695m\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:12.200188 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.200142    4559 generic.go:356] "Generic (PLEG): container finished" podID="28022dfc-ac33-46b0-ba0e-066502f8acd5" containerID="6868a61a9a84f616291d93c3379c91f0dd118c4ffd82bc5b1feaa8984002521f" exitCode=0
Oct 29 23:02:12.200640 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.200212    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr" event={"ID":"28022dfc-ac33-46b0-ba0e-066502f8acd5","Type":"ContainerDied","Data":"6868a61a9a84f616291d93c3379c91f0dd118c4ffd82bc5b1feaa8984002521f"}
Oct 29 23:02:12.200640 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.200264    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"
Oct 29 23:02:12.200640 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.200283    4559 scope.go:122] "RemoveContainer" containerID="6868a61a9a84f616291d93c3379c91f0dd118c4ffd82bc5b1feaa8984002521f"
Oct 29 23:02:12.200640 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.200272    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr" event={"ID":"28022dfc-ac33-46b0-ba0e-066502f8acd5","Type":"ContainerDied","Data":"8a4898884c9ef43fae0716b1b70285ff7fe1d5f17aac9fcf70f8b4dbcc20fb78"}
Oct 29 23:02:12.202187 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.202170    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"
Oct 29 23:02:12.202389 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.202211    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7" event={"ID":"5022bff6-abb7-4762-87b0-aa1b6f71ab70","Type":"ContainerDied","Data":"a241643a49091cbd14d692019308a4b2c85f03718bef5511d155eff7eeee898e"}
Oct 29 23:02:12.202493 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.202422    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a241643a49091cbd14d692019308a4b2c85f03718bef5511d155eff7eeee898e"
Oct 29 23:02:12.205231 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.205207    4559 generic.go:356] "Generic (PLEG): container finished" podID="dbc13347-02e3-42dd-88f6-bf1503c428ef" containerID="8661bc9a6c87fb9f5a5182d59543a3185c3c2cff22eff6d0fbf649f4996b5512" exitCode=0
Oct 29 23:02:12.205372 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.205251    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"
Oct 29 23:02:12.205372 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.205298    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv" event={"ID":"dbc13347-02e3-42dd-88f6-bf1503c428ef","Type":"ContainerDied","Data":"8661bc9a6c87fb9f5a5182d59543a3185c3c2cff22eff6d0fbf649f4996b5512"}
Oct 29 23:02:12.205372 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.205346    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv" event={"ID":"dbc13347-02e3-42dd-88f6-bf1503c428ef","Type":"ContainerDied","Data":"d1b59a1ec8929c3dbf7c86f8e17ccf0f77c311c7cf5dc75cfd8d60ffc48b9838"}
Oct 29 23:02:12.213877 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.213763    4559 scope.go:122] "RemoveContainer" containerID="6868a61a9a84f616291d93c3379c91f0dd118c4ffd82bc5b1feaa8984002521f"
Oct 29 23:02:12.218260 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:02:12.218223    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6868a61a9a84f616291d93c3379c91f0dd118c4ffd82bc5b1feaa8984002521f\": container with ID starting with 6868a61a9a84f616291d93c3379c91f0dd118c4ffd82bc5b1feaa8984002521f not found: ID does not exist" containerID="6868a61a9a84f616291d93c3379c91f0dd118c4ffd82bc5b1feaa8984002521f"
Oct 29 23:02:12.218402 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.218270    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6868a61a9a84f616291d93c3379c91f0dd118c4ffd82bc5b1feaa8984002521f"} err="failed to get container status \"6868a61a9a84f616291d93c3379c91f0dd118c4ffd82bc5b1feaa8984002521f\": rpc error: code = NotFound desc = could not find container \"6868a61a9a84f616291d93c3379c91f0dd118c4ffd82bc5b1feaa8984002521f\": container with ID starting with 6868a61a9a84f616291d93c3379c91f0dd118c4ffd82bc5b1feaa8984002521f not found: ID does not exist"
Oct 29 23:02:12.218402 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.218294    4559 scope.go:122] "RemoveContainer" containerID="8661bc9a6c87fb9f5a5182d59543a3185c3c2cff22eff6d0fbf649f4996b5512"
Oct 29 23:02:12.239740 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.239500    4559 scope.go:122] "RemoveContainer" containerID="8661bc9a6c87fb9f5a5182d59543a3185c3c2cff22eff6d0fbf649f4996b5512"
Oct 29 23:02:12.240074 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:02:12.240036    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8661bc9a6c87fb9f5a5182d59543a3185c3c2cff22eff6d0fbf649f4996b5512\": container with ID starting with 8661bc9a6c87fb9f5a5182d59543a3185c3c2cff22eff6d0fbf649f4996b5512 not found: ID does not exist" containerID="8661bc9a6c87fb9f5a5182d59543a3185c3c2cff22eff6d0fbf649f4996b5512"
Oct 29 23:02:12.240232 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.240085    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8661bc9a6c87fb9f5a5182d59543a3185c3c2cff22eff6d0fbf649f4996b5512"} err="failed to get container status \"8661bc9a6c87fb9f5a5182d59543a3185c3c2cff22eff6d0fbf649f4996b5512\": rpc error: code = NotFound desc = could not find container \"8661bc9a6c87fb9f5a5182d59543a3185c3c2cff22eff6d0fbf649f4996b5512\": container with ID starting with 8661bc9a6c87fb9f5a5182d59543a3185c3c2cff22eff6d0fbf649f4996b5512 not found: ID does not exist"
Oct 29 23:02:12.285228 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.285184    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"]
Oct 29 23:02:12.288841 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.288799    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-63e89c-744cbf57c9-k8rvr"]
Oct 29 23:02:12.323375 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.323332    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"]
Oct 29 23:02:12.327504 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.327472    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-63e89c-68b459b96-7hqcv"]
Oct 29 23:02:12.845362 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.845325    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 23:02:12.941679 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.941316    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-data\") pod \"6f118086-bc70-4d6d-874a-12ebec737d0d\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") "
Oct 29 23:02:12.941912 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.941639    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-data" pod "6f118086-bc70-4d6d-874a-12ebec737d0d" (UID: "6f118086-bc70-4d6d-874a-12ebec737d0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:12.941912 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.941881    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-workload-socket\") pod \"6f118086-bc70-4d6d-874a-12ebec737d0d\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") "
Oct 29 23:02:12.942523 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.942495    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-workload-socket" pod "6f118086-bc70-4d6d-874a-12ebec737d0d" (UID: "6f118086-bc70-4d6d-874a-12ebec737d0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:12.942725 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.942532    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-podinfo\") pod \"6f118086-bc70-4d6d-874a-12ebec737d0d\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") "
Oct 29 23:02:12.942725 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.942636    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f118086-bc70-4d6d-874a-12ebec737d0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f118086-bc70-4d6d-874a-12ebec737d0d-istiod-ca-cert\") pod \"6f118086-bc70-4d6d-874a-12ebec737d0d\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") "
Oct 29 23:02:12.943261 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.943234    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f118086-bc70-4d6d-874a-12ebec737d0d-istiod-ca-cert" pod "6f118086-bc70-4d6d-874a-12ebec737d0d" (UID: "6f118086-bc70-4d6d-874a-12ebec737d0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:12.943387 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.942667    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-envoy\") pod \"6f118086-bc70-4d6d-874a-12ebec737d0d\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") "
Oct 29 23:02:12.943477 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.943392    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-token\" (UniqueName: \"kubernetes.io/projected/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-token\") pod \"6f118086-bc70-4d6d-874a-12ebec737d0d\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") "
Oct 29 23:02:12.943477 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.943442    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-credential-socket\") pod \"6f118086-bc70-4d6d-874a-12ebec737d0d\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") "
Oct 29 23:02:12.943646 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.943483    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-workload-certs\") pod \"6f118086-bc70-4d6d-874a-12ebec737d0d\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") "
Oct 29 23:02:12.943646 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.943567    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f118086-bc70-4d6d-874a-12ebec737d0d-kube-api-access-6z8wk\" (UniqueName: \"kubernetes.io/projected/6f118086-bc70-4d6d-874a-12ebec737d0d-kube-api-access-6z8wk\") pod \"6f118086-bc70-4d6d-874a-12ebec737d0d\" (UID: \"6f118086-bc70-4d6d-874a-12ebec737d0d\") "
Oct 29 23:02:12.944230 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.944196    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:12.944230 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.944225    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f118086-bc70-4d6d-874a-12ebec737d0d-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:12.944414 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.944237    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:12.945203 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.945176    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-credential-socket" pod "6f118086-bc70-4d6d-874a-12ebec737d0d" (UID: "6f118086-bc70-4d6d-874a-12ebec737d0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:12.945665 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.945634    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-workload-certs" pod "6f118086-bc70-4d6d-874a-12ebec737d0d" (UID: "6f118086-bc70-4d6d-874a-12ebec737d0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:12.948282 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.948220    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-podinfo" pod "6f118086-bc70-4d6d-874a-12ebec737d0d" (UID: "6f118086-bc70-4d6d-874a-12ebec737d0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:12.951987 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.951947    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-envoy" pod "6f118086-bc70-4d6d-874a-12ebec737d0d" (UID: "6f118086-bc70-4d6d-874a-12ebec737d0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:12.953220 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.953190    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f118086-bc70-4d6d-874a-12ebec737d0d-kube-api-access-6z8wk" pod "6f118086-bc70-4d6d-874a-12ebec737d0d" (UID: "6f118086-bc70-4d6d-874a-12ebec737d0d"). InnerVolumeSpecName "kube-api-access-6z8wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:12.957309 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:12.957283    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-token" pod "6f118086-bc70-4d6d-874a-12ebec737d0d" (UID: "6f118086-bc70-4d6d-874a-12ebec737d0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:13.046133 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.046090    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z8wk\" (UniqueName: \"kubernetes.io/projected/6f118086-bc70-4d6d-874a-12ebec737d0d-kube-api-access-6z8wk\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:13.046133 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.046126    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:13.046133 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.046140    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:13.046402 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.046151    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f118086-bc70-4d6d-874a-12ebec737d0d-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:13.046402 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.046163    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:13.046402 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.046175    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f118086-bc70-4d6d-874a-12ebec737d0d-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:13.215958 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.215872    4559 generic.go:356] "Generic (PLEG): container finished" podID="6f118086-bc70-4d6d-874a-12ebec737d0d" containerID="59c509d1c569f337f550c077bbe52831b0e237ddf1f21a3c5a355da8444f2732" exitCode=0
Oct 29 23:02:13.215958 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.215934    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"
Oct 29 23:02:13.216499 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.215960    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn" event={"ID":"6f118086-bc70-4d6d-874a-12ebec737d0d","Type":"ContainerDied","Data":"59c509d1c569f337f550c077bbe52831b0e237ddf1f21a3c5a355da8444f2732"}
Oct 29 23:02:13.216499 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.216011    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn" event={"ID":"6f118086-bc70-4d6d-874a-12ebec737d0d","Type":"ContainerDied","Data":"17060c2b36cf73b164e124a79d165e5a5b1b0532223cc0d6e2f8004b49748737"}
Oct 29 23:02:13.216499 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.216034    4559 scope.go:122] "RemoveContainer" containerID="59c509d1c569f337f550c077bbe52831b0e237ddf1f21a3c5a355da8444f2732"
Oct 29 23:02:13.228464 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.228441    4559 scope.go:122] "RemoveContainer" containerID="9c9dd15fc4007ca1dc5545302ff96155c2433d00839fb6e349563d7097229e61"
Oct 29 23:02:13.229721 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.229700    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28022dfc-ac33-46b0-ba0e-066502f8acd5" path="/var/lib/kubelet/pods/28022dfc-ac33-46b0-ba0e-066502f8acd5/volumes"
Oct 29 23:02:13.230458 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.230433    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52d2294b-e879-4782-b9e9-526f2aeecdba" path="/var/lib/kubelet/pods/52d2294b-e879-4782-b9e9-526f2aeecdba/volumes"
Oct 29 23:02:13.231038 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.231019    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbc13347-02e3-42dd-88f6-bf1503c428ef" path="/var/lib/kubelet/pods/dbc13347-02e3-42dd-88f6-bf1503c428ef/volumes"
Oct 29 23:02:13.240491 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.240323    4559 scope.go:122] "RemoveContainer" containerID="59c509d1c569f337f550c077bbe52831b0e237ddf1f21a3c5a355da8444f2732"
Oct 29 23:02:13.240777 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:02:13.240730    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c509d1c569f337f550c077bbe52831b0e237ddf1f21a3c5a355da8444f2732\": container with ID starting with 59c509d1c569f337f550c077bbe52831b0e237ddf1f21a3c5a355da8444f2732 not found: ID does not exist" containerID="59c509d1c569f337f550c077bbe52831b0e237ddf1f21a3c5a355da8444f2732"
Oct 29 23:02:13.240777 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.240760    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c509d1c569f337f550c077bbe52831b0e237ddf1f21a3c5a355da8444f2732"} err="failed to get container status \"59c509d1c569f337f550c077bbe52831b0e237ddf1f21a3c5a355da8444f2732\": rpc error: code = NotFound desc = could not find container \"59c509d1c569f337f550c077bbe52831b0e237ddf1f21a3c5a355da8444f2732\": container with ID starting with 59c509d1c569f337f550c077bbe52831b0e237ddf1f21a3c5a355da8444f2732 not found: ID does not exist"
Oct 29 23:02:13.240777 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.240778    4559 scope.go:122] "RemoveContainer" containerID="9c9dd15fc4007ca1dc5545302ff96155c2433d00839fb6e349563d7097229e61"
Oct 29 23:02:13.241130 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:02:13.241102    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c9dd15fc4007ca1dc5545302ff96155c2433d00839fb6e349563d7097229e61\": container with ID starting with 9c9dd15fc4007ca1dc5545302ff96155c2433d00839fb6e349563d7097229e61 not found: ID does not exist" containerID="9c9dd15fc4007ca1dc5545302ff96155c2433d00839fb6e349563d7097229e61"
Oct 29 23:02:13.241237 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.241136    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c9dd15fc4007ca1dc5545302ff96155c2433d00839fb6e349563d7097229e61"} err="failed to get container status \"9c9dd15fc4007ca1dc5545302ff96155c2433d00839fb6e349563d7097229e61\": rpc error: code = NotFound desc = could not find container \"9c9dd15fc4007ca1dc5545302ff96155c2433d00839fb6e349563d7097229e61\": container with ID starting with 9c9dd15fc4007ca1dc5545302ff96155c2433d00839fb6e349563d7097229e61 not found: ID does not exist"
Oct 29 23:02:13.275146 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.275071    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"]
Oct 29 23:02:13.290086 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.290053    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn"]
Oct 29 23:02:13.695318 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:02:13.695279    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:02:13.791417 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.790944    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:02:13.791417 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:13.791042    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-75b68d-openshift-default-5f58db85f7-74lsn" podUID="6f118086-bc70-4d6d-874a-12ebec737d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:02:15.124869 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.124794    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"]
Oct 29 23:02:15.125746 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.125720    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6f118086-bc70-4d6d-874a-12ebec737d0d" containerName="istio-proxy"
Oct 29 23:02:15.125746 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.125746    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f118086-bc70-4d6d-874a-12ebec737d0d" containerName="istio-proxy"
Oct 29 23:02:15.126004 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.125768    4559 cpu_manager.go:414] "containerMap: removing container" podUID="dbc13347-02e3-42dd-88f6-bf1503c428ef" containerName="payload-processing"
Oct 29 23:02:15.126004 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.125775    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbc13347-02e3-42dd-88f6-bf1503c428ef" containerName="payload-processing"
Oct 29 23:02:15.126004 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.125792    4559 cpu_manager.go:414] "containerMap: removing container" podUID="28022dfc-ac33-46b0-ba0e-066502f8acd5" containerName="payload-pre-processing"
Oct 29 23:02:15.126004 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.125800    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28022dfc-ac33-46b0-ba0e-066502f8acd5" containerName="payload-pre-processing"
Oct 29 23:02:15.126004 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.125851    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6f118086-bc70-4d6d-874a-12ebec737d0d" containerName="istio-proxy"
Oct 29 23:02:15.126004 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.125859    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f118086-bc70-4d6d-874a-12ebec737d0d" containerName="istio-proxy"
Oct 29 23:02:15.126004 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.125890    4559 cpu_manager.go:414] "containerMap: removing container" podUID="52d2294b-e879-4782-b9e9-526f2aeecdba" containerName="istio-proxy"
Oct 29 23:02:15.126004 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.125897    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d2294b-e879-4782-b9e9-526f2aeecdba" containerName="istio-proxy"
Oct 29 23:02:15.126004 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.125918    4559 cpu_manager.go:414] "containerMap: removing container" podUID="52d2294b-e879-4782-b9e9-526f2aeecdba" containerName="istio-proxy"
Oct 29 23:02:15.126004 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.125924    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d2294b-e879-4782-b9e9-526f2aeecdba" containerName="istio-proxy"
Oct 29 23:02:15.126004 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.125953    4559 cpu_manager.go:414] "containerMap: removing container" podUID="5022bff6-abb7-4762-87b0-aa1b6f71ab70" containerName="revoke-keys"
Oct 29 23:02:15.126004 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.125962    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5022bff6-abb7-4762-87b0-aa1b6f71ab70" containerName="revoke-keys"
Oct 29 23:02:15.126618 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.126124    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="5022bff6-abb7-4762-87b0-aa1b6f71ab70" containerName="revoke-keys"
Oct 29 23:02:15.126618 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.126142    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f118086-bc70-4d6d-874a-12ebec737d0d" containerName="istio-proxy"
Oct 29 23:02:15.126618 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.126154    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbc13347-02e3-42dd-88f6-bf1503c428ef" containerName="payload-processing"
Oct 29 23:02:15.126618 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.126164    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="28022dfc-ac33-46b0-ba0e-066502f8acd5" containerName="payload-pre-processing"
Oct 29 23:02:15.126618 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.126181    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f118086-bc70-4d6d-874a-12ebec737d0d" containerName="istio-proxy"
Oct 29 23:02:15.126618 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.126196    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d2294b-e879-4782-b9e9-526f2aeecdba" containerName="istio-proxy"
Oct 29 23:02:15.126618 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.126206    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d2294b-e879-4782-b9e9-526f2aeecdba" containerName="istio-proxy"
Oct 29 23:02:15.132076 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.132049    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"
Oct 29 23:02:15.151664 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.151631    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"]
Oct 29 23:02:15.163748 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.158151    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:02:15.163748 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.158159    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:02:15.239005 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.238962    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f118086-bc70-4d6d-874a-12ebec737d0d" path="/var/lib/kubelet/pods/6f118086-bc70-4d6d-874a-12ebec737d0d/volumes"
Oct 29 23:02:15.282059 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.281994    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98b573a-0354-408a-a865-67705b3e070d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp\" (UID: \"c98b573a-0354-408a-a865-67705b3e070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"
Oct 29 23:02:15.282279 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.282090    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98b573a-0354-408a-a865-67705b3e070d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp\" (UID: \"c98b573a-0354-408a-a865-67705b3e070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"
Oct 29 23:02:15.282279 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.282252    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pfv2\" (UniqueName: \"kubernetes.io/projected/c98b573a-0354-408a-a865-67705b3e070d-kube-api-access-7pfv2\") pod \"maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp\" (UID: \"c98b573a-0354-408a-a865-67705b3e070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"
Oct 29 23:02:15.383352 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.383247    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98b573a-0354-408a-a865-67705b3e070d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp\" (UID: \"c98b573a-0354-408a-a865-67705b3e070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"
Oct 29 23:02:15.383352 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.383351    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pfv2\" (UniqueName: \"kubernetes.io/projected/c98b573a-0354-408a-a865-67705b3e070d-kube-api-access-7pfv2\") pod \"maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp\" (UID: \"c98b573a-0354-408a-a865-67705b3e070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"
Oct 29 23:02:15.383645 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.383499    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98b573a-0354-408a-a865-67705b3e070d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp\" (UID: \"c98b573a-0354-408a-a865-67705b3e070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"
Oct 29 23:02:15.384135 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.384114    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98b573a-0354-408a-a865-67705b3e070d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp\" (UID: \"c98b573a-0354-408a-a865-67705b3e070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"
Oct 29 23:02:15.385951 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.385925    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98b573a-0354-408a-a865-67705b3e070d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp\" (UID: \"c98b573a-0354-408a-a865-67705b3e070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"
Oct 29 23:02:15.405970 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.405939    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pfv2\" (UniqueName: \"kubernetes.io/projected/c98b573a-0354-408a-a865-67705b3e070d-kube-api-access-7pfv2\") pod \"maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp\" (UID: \"c98b573a-0354-408a-a865-67705b3e070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"
Oct 29 23:02:15.448917 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.448882    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"
Oct 29 23:02:15.800471 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:15.800443    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"]
Oct 29 23:02:15.801427 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:02:15.801402    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98b573a_0354_408a_a865_67705b3e070d.slice/crio-8f334808e1a734f9248c6d6597ef44430d13dd4272988804025b3baf640ed11b WatchSource:0}: Error finding container 8f334808e1a734f9248c6d6597ef44430d13dd4272988804025b3baf640ed11b: Status 404 returned error can't find the container with id 8f334808e1a734f9248c6d6597ef44430d13dd4272988804025b3baf640ed11b
Oct 29 23:02:16.253698 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:16.253655    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp" event={"ID":"c98b573a-0354-408a-a865-67705b3e070d","Type":"ContainerStarted","Data":"3bfe397013fbf882777a3302c6ddab0d77e5ebe4a45e02f7025ece4402cdb4e7"}
Oct 29 23:02:16.254268 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:16.253729    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp" event={"ID":"c98b573a-0354-408a-a865-67705b3e070d","Type":"ContainerStarted","Data":"8f334808e1a734f9248c6d6597ef44430d13dd4272988804025b3baf640ed11b"}
Oct 29 23:02:16.296186 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:16.296116    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp" podStartSLOduration=1.296096031 podStartE2EDuration="1.296096031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:16.291883717 +0000 UTC m=+2875.203327271" watchObservedRunningTime="2026-10-29 23:02:16.296096031 +0000 UTC m=+2875.207539586"
Oct 29 23:02:17.264226 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:17.264192    4559 generic.go:356] "Generic (PLEG): container finished" podID="c98b573a-0354-408a-a865-67705b3e070d" containerID="3bfe397013fbf882777a3302c6ddab0d77e5ebe4a45e02f7025ece4402cdb4e7" exitCode=0
Oct 29 23:02:17.264709 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:17.264274    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp" event={"ID":"c98b573a-0354-408a-a865-67705b3e070d","Type":"ContainerDied","Data":"3bfe397013fbf882777a3302c6ddab0d77e5ebe4a45e02f7025ece4402cdb4e7"}
Oct 29 23:02:18.577218 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:18.577189    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"
Oct 29 23:02:18.579153 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:18.579123    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"]
Oct 29 23:02:18.613632 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:18.613593    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"]
Oct 29 23:02:18.716873 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:18.714234    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98b573a-0354-408a-a865-67705b3e070d-kube-api-access-7pfv2\" (UniqueName: \"kubernetes.io/projected/c98b573a-0354-408a-a865-67705b3e070d-kube-api-access-7pfv2\") pod \"c98b573a-0354-408a-a865-67705b3e070d\" (UID: \"c98b573a-0354-408a-a865-67705b3e070d\") "
Oct 29 23:02:18.716873 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:18.714385    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c98b573a-0354-408a-a865-67705b3e070d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98b573a-0354-408a-a865-67705b3e070d-maas-api-openshift-ca\") pod \"c98b573a-0354-408a-a865-67705b3e070d\" (UID: \"c98b573a-0354-408a-a865-67705b3e070d\") "
Oct 29 23:02:18.716873 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:18.714443    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c98b573a-0354-408a-a865-67705b3e070d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98b573a-0354-408a-a865-67705b3e070d-maas-api-xks-ca\") pod \"c98b573a-0354-408a-a865-67705b3e070d\" (UID: \"c98b573a-0354-408a-a865-67705b3e070d\") "
Oct 29 23:02:18.716873 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:18.715140    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98b573a-0354-408a-a865-67705b3e070d-maas-api-openshift-ca" pod "c98b573a-0354-408a-a865-67705b3e070d" (UID: "c98b573a-0354-408a-a865-67705b3e070d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:18.716873 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:18.715363    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98b573a-0354-408a-a865-67705b3e070d-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.718381 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:18.718354    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98b573a-0354-408a-a865-67705b3e070d-kube-api-access-7pfv2" pod "c98b573a-0354-408a-a865-67705b3e070d" (UID: "c98b573a-0354-408a-a865-67705b3e070d"). InnerVolumeSpecName "kube-api-access-7pfv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:18.721204 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:18.721177    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c98b573a-0354-408a-a865-67705b3e070d-maas-api-xks-ca" pod "c98b573a-0354-408a-a865-67705b3e070d" (UID: "c98b573a-0354-408a-a865-67705b3e070d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:18.858111 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:18.858017    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98b573a-0354-408a-a865-67705b3e070d-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.858342 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:18.858327    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pfv2\" (UniqueName: \"kubernetes.io/projected/c98b573a-0354-408a-a865-67705b3e070d-kube-api-access-7pfv2\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:19.283266 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.283231    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"
Oct 29 23:02:19.283266 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.283253    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp" event={"ID":"c98b573a-0354-408a-a865-67705b3e070d","Type":"ContainerDied","Data":"8f334808e1a734f9248c6d6597ef44430d13dd4272988804025b3baf640ed11b"}
Oct 29 23:02:19.283507 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.283307    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f334808e1a734f9248c6d6597ef44430d13dd4272988804025b3baf640ed11b"
Oct 29 23:02:19.434905 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.427969    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"]
Oct 29 23:02:19.439829 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.436967    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c98b573a-0354-408a-a865-67705b3e070d" containerName="revoke-keys"
Oct 29 23:02:19.439829 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.436999    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c98b573a-0354-408a-a865-67705b3e070d" containerName="revoke-keys"
Oct 29 23:02:19.439829 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.437266    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c98b573a-0354-408a-a865-67705b3e070d" containerName="revoke-keys"
Oct 29 23:02:19.448793 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.448760    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"
Oct 29 23:02:19.478313 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.470851    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"]
Oct 29 23:02:19.478607 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.471906    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6gn\" (UniqueName: \"kubernetes.io/projected/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-kube-api-access-7l6gn\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg\" (UID: \"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"
Oct 29 23:02:19.479209 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.479177    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg\" (UID: \"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"
Oct 29 23:02:19.479505 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.479487    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg\" (UID: \"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"
Oct 29 23:02:19.502475 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.501709    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:02:19.502475 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.501871    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:02:19.580986 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.580934    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg\" (UID: \"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"
Oct 29 23:02:19.581478 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.581094    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg\" (UID: \"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"
Oct 29 23:02:19.581478 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.581243    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6gn\" (UniqueName: \"kubernetes.io/projected/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-kube-api-access-7l6gn\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg\" (UID: \"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"
Oct 29 23:02:19.581801 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.581778    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg\" (UID: \"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"
Oct 29 23:02:19.583645 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.583616    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg\" (UID: \"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"
Oct 29 23:02:19.600760 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.600731    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6gn\" (UniqueName: \"kubernetes.io/projected/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-kube-api-access-7l6gn\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg\" (UID: \"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"
Oct 29 23:02:19.810017 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:19.809905    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"
Oct 29 23:02:20.119692 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:02:20.119645    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f6c0c6_84c4_424e_a740_9e2dd5fe332c.slice/crio-03eaa5467e5faefc7c69aa4810af64ac071926028d8bae7c502ad5d74f99d81c WatchSource:0}: Error finding container 03eaa5467e5faefc7c69aa4810af64ac071926028d8bae7c502ad5d74f99d81c: Status 404 returned error can't find the container with id 03eaa5467e5faefc7c69aa4810af64ac071926028d8bae7c502ad5d74f99d81c
Oct 29 23:02:20.120063 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:20.120040    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"]
Oct 29 23:02:20.297896 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:20.297850    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg" event={"ID":"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c","Type":"ContainerStarted","Data":"03eaa5467e5faefc7c69aa4810af64ac071926028d8bae7c502ad5d74f99d81c"}
Oct 29 23:02:21.307948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:21.307913    4559 generic.go:356] "Generic (PLEG): container finished" podID="d8f6c0c6-84c4-424e-a740-9e2dd5fe332c" containerID="ca26215ac89336825eb6cb8bc185a563a6426909b886d5d0acc47a3d4ab4b696" exitCode=0
Oct 29 23:02:21.308377 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:21.307999    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg" event={"ID":"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c","Type":"ContainerDied","Data":"ca26215ac89336825eb6cb8bc185a563a6426909b886d5d0acc47a3d4ab4b696"}
Oct 29 23:02:22.487593 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:22.487564    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"
Oct 29 23:02:22.519998 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:22.519962    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-kube-api-access-7l6gn\" (UniqueName: \"kubernetes.io/projected/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-kube-api-access-7l6gn\") pod \"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c\" (UID: \"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c\") "
Oct 29 23:02:22.520249 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:22.520226    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-maas-api-openshift-ca\") pod \"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c\" (UID: \"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c\") "
Oct 29 23:02:22.520311 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:22.520259    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-maas-api-xks-ca\") pod \"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c\" (UID: \"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c\") "
Oct 29 23:02:22.520569 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:22.520542    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-maas-api-openshift-ca" pod "d8f6c0c6-84c4-424e-a740-9e2dd5fe332c" (UID: "d8f6c0c6-84c4-424e-a740-9e2dd5fe332c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:22.522319 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:22.522294    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-kube-api-access-7l6gn" pod "d8f6c0c6-84c4-424e-a740-9e2dd5fe332c" (UID: "d8f6c0c6-84c4-424e-a740-9e2dd5fe332c"). InnerVolumeSpecName "kube-api-access-7l6gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:22.522454 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:22.522429    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-maas-api-xks-ca" pod "d8f6c0c6-84c4-424e-a740-9e2dd5fe332c" (UID: "d8f6c0c6-84c4-424e-a740-9e2dd5fe332c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:22.621875 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:22.621779    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:22.621875 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:22.621853    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:22.621875 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:22.621864    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l6gn\" (UniqueName: \"kubernetes.io/projected/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c-kube-api-access-7l6gn\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:23.326436 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:23.326405    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"
Oct 29 23:02:23.326620 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:23.326442    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg" event={"ID":"d8f6c0c6-84c4-424e-a740-9e2dd5fe332c","Type":"ContainerDied","Data":"03eaa5467e5faefc7c69aa4810af64ac071926028d8bae7c502ad5d74f99d81c"}
Oct 29 23:02:23.326620 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:23.326487    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03eaa5467e5faefc7c69aa4810af64ac071926028d8bae7c502ad5d74f99d81c"
Oct 29 23:02:23.579693 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:23.579582    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn" podUID="ec5a59ee-c4bb-4457-8db1-9d098bd4147e" containerName="payload-processing" containerID="cri-o://c26327bd9adfcced2c9000b49138ad53851abebd5f3e711a1172975dea493efe" gracePeriod=25
Oct 29 23:02:23.614425 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:23.614383    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx" podUID="655d0083-da33-47ed-bb81-a3433a69da00" containerName="payload-pre-processing" containerID="cri-o://2d4c5e1e7aad9fcb8afb2558b79b0fbb4fc09413189ba94f1f00493c36f85e59" gracePeriod=25
Oct 29 23:02:24.541551 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:24.541517    4559 scope.go:122] "RemoveContainer" containerID="c14e601cf13401fd733920ccd520cd01cca892f63c229a14cb50b21b4cad9afb"
Oct 29 23:02:24.554195 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:24.554165    4559 scope.go:122] "RemoveContainer" containerID="3933d302c9f33235ca7fa5b68cab508498d15014e84cb6952a796c14ffcbcbbf"
Oct 29 23:02:29.659844 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:29.659795    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"]
Oct 29 23:02:29.676097 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:29.676045    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"]
Oct 29 23:02:31.284241 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:31.284199    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"]
Oct 29 23:02:31.296705 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:31.296671    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-920032-zltbg"]
Oct 29 23:02:32.778715 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:32.778673    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn" podUID="ec5a59ee-c4bb-4457-8db1-9d098bd4147e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:02:32.779179 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:32.778690    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx" podUID="655d0083-da33-47ed-bb81-a3433a69da00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:02:33.228440 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:33.228402    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f6c0c6-84c4-424e-a740-9e2dd5fe332c" path="/var/lib/kubelet/pods/d8f6c0c6-84c4-424e-a740-9e2dd5fe332c/volumes"
Oct 29 23:02:34.664053 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:34.663992    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765" podUID="fd73261d-a0be-4df4-99f6-73d2ea1c425a" containerName="payload-processing" containerID="cri-o://b059f7beee73e4cdbf97ac042eacdeeb9a69324b82b9965eb6594230bf048b0b" gracePeriod=25
Oct 29 23:02:34.677332 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:34.677180    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62" podUID="290fd117-eaa7-4003-859d-3dd8ad075e05" containerName="payload-pre-processing" containerID="cri-o://966dcdad4418935fdcb245ba3a4eea708534bd4b09a0e78bf9af2834381f8aa1" gracePeriod=25
Oct 29 23:02:35.667852 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:35.667757    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"]
Oct 29 23:02:35.673318 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:35.673286    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-eb02df-lpmw7"]
Oct 29 23:02:37.229085 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:37.229050    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5022bff6-abb7-4762-87b0-aa1b6f71ab70" path="/var/lib/kubelet/pods/5022bff6-abb7-4762-87b0-aa1b6f71ab70/volumes"
Oct 29 23:02:38.087462 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:38.087419    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"]
Oct 29 23:02:38.087792 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:38.087734    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp" podUID="700d065a-5535-41fb-bbc6-ba2fc54cd0a4" containerName="istio-proxy" containerID="cri-o://8abd15896824d6311f1fd8283b3f3979b899050a7a6404509f65749d4dec5d4f" gracePeriod=30
Oct 29 23:02:38.880034 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:38.879996    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:02:42.473713 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:42.473674    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"]
Oct 29 23:02:42.474359 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:42.474324    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv" podUID="d9022f52-574d-478f-ae49-29732f126feb" containerName="istio-proxy" containerID="cri-o://9109a8eaded4ef240af04cd37dc62d3eb19ebd6c8978bc90290addbdcb385071" gracePeriod=30
Oct 29 23:02:42.783711 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:42.783611    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx" podUID="655d0083-da33-47ed-bb81-a3433a69da00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:02:42.784330 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:42.784291    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn" podUID="ec5a59ee-c4bb-4457-8db1-9d098bd4147e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:02:42.853775 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:42.853732    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"]
Oct 29 23:02:42.854058 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:42.854016    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw" podUID="d5789ad1-f64c-4eb8-9a38-49da59afb179" containerName="payload-processing" containerID="cri-o://1c412d2394b6939b47f17669f359f9b4fbba61791a651d42bb5660c2e6acb45f" gracePeriod=30
Oct 29 23:02:43.317934 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.300558    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"
Oct 29 23:02:43.471833 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.466049    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5789ad1-f64c-4eb8-9a38-49da59afb179-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5789ad1-f64c-4eb8-9a38-49da59afb179-plugins-config-volume\") pod \"d5789ad1-f64c-4eb8-9a38-49da59afb179\" (UID: \"d5789ad1-f64c-4eb8-9a38-49da59afb179\") "
Oct 29 23:02:43.471833 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.466112    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5789ad1-f64c-4eb8-9a38-49da59afb179-kube-api-access-rcjqq\" (UniqueName: \"kubernetes.io/projected/d5789ad1-f64c-4eb8-9a38-49da59afb179-kube-api-access-rcjqq\") pod \"d5789ad1-f64c-4eb8-9a38-49da59afb179\" (UID: \"d5789ad1-f64c-4eb8-9a38-49da59afb179\") "
Oct 29 23:02:43.471833 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.467511    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5789ad1-f64c-4eb8-9a38-49da59afb179-plugins-config-volume" pod "d5789ad1-f64c-4eb8-9a38-49da59afb179" (UID: "d5789ad1-f64c-4eb8-9a38-49da59afb179"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:43.484031 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.483983    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5789ad1-f64c-4eb8-9a38-49da59afb179-kube-api-access-rcjqq" pod "d5789ad1-f64c-4eb8-9a38-49da59afb179" (UID: "d5789ad1-f64c-4eb8-9a38-49da59afb179"). InnerVolumeSpecName "kube-api-access-rcjqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:43.496229 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.495124    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"]
Oct 29 23:02:43.502094 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.502056    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85862e-s8vcp"]
Oct 29 23:02:43.599529 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.575462    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 23:02:43.599529 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.577640    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5789ad1-f64c-4eb8-9a38-49da59afb179-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.599529 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.577759    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcjqq\" (UniqueName: \"kubernetes.io/projected/d5789ad1-f64c-4eb8-9a38-49da59afb179-kube-api-access-rcjqq\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.599529 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.583471    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62" podUID="290fd117-eaa7-4003-859d-3dd8ad075e05" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:02:43.599529 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.585061    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765" podUID="fd73261d-a0be-4df4-99f6-73d2ea1c425a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:02:43.643708 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.643631    4559 generic.go:356] "Generic (PLEG): container finished" podID="d5789ad1-f64c-4eb8-9a38-49da59afb179" containerID="1c412d2394b6939b47f17669f359f9b4fbba61791a651d42bb5660c2e6acb45f" exitCode=0
Oct 29 23:02:43.643917 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.643767    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw" event={"ID":"d5789ad1-f64c-4eb8-9a38-49da59afb179","Type":"ContainerDied","Data":"1c412d2394b6939b47f17669f359f9b4fbba61791a651d42bb5660c2e6acb45f"}
Oct 29 23:02:43.643917 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.643848    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw" event={"ID":"d5789ad1-f64c-4eb8-9a38-49da59afb179","Type":"ContainerDied","Data":"30de434d54c8524486d85e0d481e2ec21baf2cd2474d2068dfc8aafb20791efc"}
Oct 29 23:02:43.643917 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.643883    4559 scope.go:122] "RemoveContainer" containerID="1c412d2394b6939b47f17669f359f9b4fbba61791a651d42bb5660c2e6acb45f"
Oct 29 23:02:43.644452 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.644129    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"
Oct 29 23:02:43.649502 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.649429    4559 generic.go:356] "Generic (PLEG): container finished" podID="700d065a-5535-41fb-bbc6-ba2fc54cd0a4" containerID="8abd15896824d6311f1fd8283b3f3979b899050a7a6404509f65749d4dec5d4f" exitCode=0
Oct 29 23:02:43.649664 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.649522    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp" event={"ID":"700d065a-5535-41fb-bbc6-ba2fc54cd0a4","Type":"ContainerDied","Data":"8abd15896824d6311f1fd8283b3f3979b899050a7a6404509f65749d4dec5d4f"}
Oct 29 23:02:43.649664 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.649572    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp" event={"ID":"700d065a-5535-41fb-bbc6-ba2fc54cd0a4","Type":"ContainerDied","Data":"a3cf1e8a783a5e9e185f04ba94ae16a6fe1c905a883377520a200f3920fb3d7d"}
Oct 29 23:02:43.649833 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.649666    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"
Oct 29 23:02:43.658908 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.658869    4559 generic.go:356] "Generic (PLEG): container finished" podID="ec5a59ee-c4bb-4457-8db1-9d098bd4147e" containerID="c26327bd9adfcced2c9000b49138ad53851abebd5f3e711a1172975dea493efe" exitCode=0
Oct 29 23:02:43.659074 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.658951    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn" event={"ID":"ec5a59ee-c4bb-4457-8db1-9d098bd4147e","Type":"ContainerDied","Data":"c26327bd9adfcced2c9000b49138ad53851abebd5f3e711a1172975dea493efe"}
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.678276    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-podinfo\") pod \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") "
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.678330    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-data\") pod \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") "
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.678464    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-kube-api-access-gbg6h\" (UniqueName: \"kubernetes.io/projected/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-kube-api-access-gbg6h\") pod \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") "
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.678499    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-envoy\") pod \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") "
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.678543    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-token\" (UniqueName: \"kubernetes.io/projected/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-token\") pod \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") "
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.678613    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-workload-socket\") pod \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") "
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.678690    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istiod-ca-cert\") pod \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") "
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.678722    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-workload-certs\") pod \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") "
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.679746    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-data" pod "700d065a-5535-41fb-bbc6-ba2fc54cd0a4" (UID: "700d065a-5535-41fb-bbc6-ba2fc54cd0a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.680171    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-workload-socket" pod "700d065a-5535-41fb-bbc6-ba2fc54cd0a4" (UID: "700d065a-5535-41fb-bbc6-ba2fc54cd0a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.680556    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istiod-ca-cert" pod "700d065a-5535-41fb-bbc6-ba2fc54cd0a4" (UID: "700d065a-5535-41fb-bbc6-ba2fc54cd0a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.680665    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-credential-socket\") pod \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\" (UID: \"700d065a-5535-41fb-bbc6-ba2fc54cd0a4\") "
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.681369    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.681394    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.681409    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.681799    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-workload-certs" pod "700d065a-5535-41fb-bbc6-ba2fc54cd0a4" (UID: "700d065a-5535-41fb-bbc6-ba2fc54cd0a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.681885    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-credential-socket" pod "700d065a-5535-41fb-bbc6-ba2fc54cd0a4" (UID: "700d065a-5535-41fb-bbc6-ba2fc54cd0a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:43.687892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.687199    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-podinfo" pod "700d065a-5535-41fb-bbc6-ba2fc54cd0a4" (UID: "700d065a-5535-41fb-bbc6-ba2fc54cd0a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:43.689711 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.689679    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-envoy" pod "700d065a-5535-41fb-bbc6-ba2fc54cd0a4" (UID: "700d065a-5535-41fb-bbc6-ba2fc54cd0a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:43.689949 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.689926    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-token" pod "700d065a-5535-41fb-bbc6-ba2fc54cd0a4" (UID: "700d065a-5535-41fb-bbc6-ba2fc54cd0a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:43.695029 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.694984    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-kube-api-access-gbg6h" pod "700d065a-5535-41fb-bbc6-ba2fc54cd0a4" (UID: "700d065a-5535-41fb-bbc6-ba2fc54cd0a4"). InnerVolumeSpecName "kube-api-access-gbg6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:43.706846 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:02:43.706785    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:02:43.745065 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.742386    4559 scope.go:122] "RemoveContainer" containerID="1c412d2394b6939b47f17669f359f9b4fbba61791a651d42bb5660c2e6acb45f"
Oct 29 23:02:43.748518 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:02:43.748335    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c412d2394b6939b47f17669f359f9b4fbba61791a651d42bb5660c2e6acb45f\": container with ID starting with 1c412d2394b6939b47f17669f359f9b4fbba61791a651d42bb5660c2e6acb45f not found: ID does not exist" containerID="1c412d2394b6939b47f17669f359f9b4fbba61791a651d42bb5660c2e6acb45f"
Oct 29 23:02:43.748518 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.748385    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c412d2394b6939b47f17669f359f9b4fbba61791a651d42bb5660c2e6acb45f"} err="failed to get container status \"1c412d2394b6939b47f17669f359f9b4fbba61791a651d42bb5660c2e6acb45f\": rpc error: code = NotFound desc = could not find container \"1c412d2394b6939b47f17669f359f9b4fbba61791a651d42bb5660c2e6acb45f\": container with ID starting with 1c412d2394b6939b47f17669f359f9b4fbba61791a651d42bb5660c2e6acb45f not found: ID does not exist"
Oct 29 23:02:43.748518 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.748414    4559 scope.go:122] "RemoveContainer" containerID="8abd15896824d6311f1fd8283b3f3979b899050a7a6404509f65749d4dec5d4f"
Oct 29 23:02:43.754081 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.754015    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"]
Oct 29 23:02:43.762404 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.762037    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85862e-69678cb8f6-hkxqw"]
Oct 29 23:02:43.772993 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.772086    4559 scope.go:122] "RemoveContainer" containerID="3877bfc2b81445c92aea7c65de6a4c43f67e2b5e4febeea6a8e8d4f8dd9531a8"
Oct 29 23:02:43.782403 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.782307    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbg6h\" (UniqueName: \"kubernetes.io/projected/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-kube-api-access-gbg6h\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.782403 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.782335    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.782403 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.782349    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.782403 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.782357    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.782403 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.782366    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.782403 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.782376    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700d065a-5535-41fb-bbc6-ba2fc54cd0a4-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.794451 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.794403    4559 scope.go:122] "RemoveContainer" containerID="8abd15896824d6311f1fd8283b3f3979b899050a7a6404509f65749d4dec5d4f"
Oct 29 23:02:43.794953 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:02:43.794922    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8abd15896824d6311f1fd8283b3f3979b899050a7a6404509f65749d4dec5d4f\": container with ID starting with 8abd15896824d6311f1fd8283b3f3979b899050a7a6404509f65749d4dec5d4f not found: ID does not exist" containerID="8abd15896824d6311f1fd8283b3f3979b899050a7a6404509f65749d4dec5d4f"
Oct 29 23:02:43.795098 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.794969    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8abd15896824d6311f1fd8283b3f3979b899050a7a6404509f65749d4dec5d4f"} err="failed to get container status \"8abd15896824d6311f1fd8283b3f3979b899050a7a6404509f65749d4dec5d4f\": rpc error: code = NotFound desc = could not find container \"8abd15896824d6311f1fd8283b3f3979b899050a7a6404509f65749d4dec5d4f\": container with ID starting with 8abd15896824d6311f1fd8283b3f3979b899050a7a6404509f65749d4dec5d4f not found: ID does not exist"
Oct 29 23:02:43.795098 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.794994    4559 scope.go:122] "RemoveContainer" containerID="3877bfc2b81445c92aea7c65de6a4c43f67e2b5e4febeea6a8e8d4f8dd9531a8"
Oct 29 23:02:43.795336 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:02:43.795308    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3877bfc2b81445c92aea7c65de6a4c43f67e2b5e4febeea6a8e8d4f8dd9531a8\": container with ID starting with 3877bfc2b81445c92aea7c65de6a4c43f67e2b5e4febeea6a8e8d4f8dd9531a8 not found: ID does not exist" containerID="3877bfc2b81445c92aea7c65de6a4c43f67e2b5e4febeea6a8e8d4f8dd9531a8"
Oct 29 23:02:43.795446 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.795345    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3877bfc2b81445c92aea7c65de6a4c43f67e2b5e4febeea6a8e8d4f8dd9531a8"} err="failed to get container status \"3877bfc2b81445c92aea7c65de6a4c43f67e2b5e4febeea6a8e8d4f8dd9531a8\": rpc error: code = NotFound desc = could not find container \"3877bfc2b81445c92aea7c65de6a4c43f67e2b5e4febeea6a8e8d4f8dd9531a8\": container with ID starting with 3877bfc2b81445c92aea7c65de6a4c43f67e2b5e4febeea6a8e8d4f8dd9531a8 not found: ID does not exist"
Oct 29 23:02:43.812313 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.812283    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"
Oct 29 23:02:43.981740 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.981712    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"
Oct 29 23:02:43.989318 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.989298    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec5a59ee-c4bb-4457-8db1-9d098bd4147e-kube-api-access-h8ppz\" (UniqueName: \"kubernetes.io/projected/ec5a59ee-c4bb-4457-8db1-9d098bd4147e-kube-api-access-h8ppz\") pod \"ec5a59ee-c4bb-4457-8db1-9d098bd4147e\" (UID: \"ec5a59ee-c4bb-4457-8db1-9d098bd4147e\") "
Oct 29 23:02:43.989457 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.989329    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec5a59ee-c4bb-4457-8db1-9d098bd4147e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5a59ee-c4bb-4457-8db1-9d098bd4147e-plugins-config-volume\") pod \"ec5a59ee-c4bb-4457-8db1-9d098bd4147e\" (UID: \"ec5a59ee-c4bb-4457-8db1-9d098bd4147e\") "
Oct 29 23:02:43.989739 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.989713    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec5a59ee-c4bb-4457-8db1-9d098bd4147e-plugins-config-volume" pod "ec5a59ee-c4bb-4457-8db1-9d098bd4147e" (UID: "ec5a59ee-c4bb-4457-8db1-9d098bd4147e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:43.989901 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.989871    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5a59ee-c4bb-4457-8db1-9d098bd4147e-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.991570 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:43.991551    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5a59ee-c4bb-4457-8db1-9d098bd4147e-kube-api-access-h8ppz" pod "ec5a59ee-c4bb-4457-8db1-9d098bd4147e" (UID: "ec5a59ee-c4bb-4457-8db1-9d098bd4147e"). InnerVolumeSpecName "kube-api-access-h8ppz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:44.019615 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.019579    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"]
Oct 29 23:02:44.030261 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.029297    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-q8qxp"]
Oct 29 23:02:44.091270 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.091237    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655d0083-da33-47ed-bb81-a3433a69da00-kube-api-access-9wt7v\" (UniqueName: \"kubernetes.io/projected/655d0083-da33-47ed-bb81-a3433a69da00-kube-api-access-9wt7v\") pod \"655d0083-da33-47ed-bb81-a3433a69da00\" (UID: \"655d0083-da33-47ed-bb81-a3433a69da00\") "
Oct 29 23:02:44.091618 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.091591    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/655d0083-da33-47ed-bb81-a3433a69da00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655d0083-da33-47ed-bb81-a3433a69da00-plugins-config-volume\") pod \"655d0083-da33-47ed-bb81-a3433a69da00\" (UID: \"655d0083-da33-47ed-bb81-a3433a69da00\") "
Oct 29 23:02:44.092185 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.092158    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8ppz\" (UniqueName: \"kubernetes.io/projected/ec5a59ee-c4bb-4457-8db1-9d098bd4147e-kube-api-access-h8ppz\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:44.092377 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.092192    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655d0083-da33-47ed-bb81-a3433a69da00-plugins-config-volume" pod "655d0083-da33-47ed-bb81-a3433a69da00" (UID: "655d0083-da33-47ed-bb81-a3433a69da00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:44.094349 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.094324    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655d0083-da33-47ed-bb81-a3433a69da00-kube-api-access-9wt7v" pod "655d0083-da33-47ed-bb81-a3433a69da00" (UID: "655d0083-da33-47ed-bb81-a3433a69da00"). InnerVolumeSpecName "kube-api-access-9wt7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:44.193688 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.193652    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655d0083-da33-47ed-bb81-a3433a69da00-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:44.193688 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.193686    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wt7v\" (UniqueName: \"kubernetes.io/projected/655d0083-da33-47ed-bb81-a3433a69da00-kube-api-access-9wt7v\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:44.668890 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.668855    4559 generic.go:356] "Generic (PLEG): container finished" podID="655d0083-da33-47ed-bb81-a3433a69da00" containerID="2d4c5e1e7aad9fcb8afb2558b79b0fbb4fc09413189ba94f1f00493c36f85e59" exitCode=0
Oct 29 23:02:44.669423 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.668930    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"
Oct 29 23:02:44.669423 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.668939    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx" event={"ID":"655d0083-da33-47ed-bb81-a3433a69da00","Type":"ContainerDied","Data":"2d4c5e1e7aad9fcb8afb2558b79b0fbb4fc09413189ba94f1f00493c36f85e59"}
Oct 29 23:02:44.669423 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.668998    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx" event={"ID":"655d0083-da33-47ed-bb81-a3433a69da00","Type":"ContainerDied","Data":"a8199cdf70821b3cb2cf9f298899ef52b0ccc8c44239c512301d24323e8b2c8e"}
Oct 29 23:02:44.669423 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.669032    4559 scope.go:122] "RemoveContainer" containerID="2d4c5e1e7aad9fcb8afb2558b79b0fbb4fc09413189ba94f1f00493c36f85e59"
Oct 29 23:02:44.673727 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.673703    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"
Oct 29 23:02:44.673727 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.673702    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn" event={"ID":"ec5a59ee-c4bb-4457-8db1-9d098bd4147e","Type":"ContainerDied","Data":"a98cf20985eef3cfeee201ffe17034301006822be51be822baf867fd25a00041"}
Oct 29 23:02:44.676731 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.676705    4559 generic.go:356] "Generic (PLEG): container finished" podID="b302c38f-92a7-4243-868b-3c3a04798313" containerID="acc784ea146aaa70d158583b457a8fe15e243934948b567fb23c07c92b8c8c42" exitCode=0
Oct 29 23:02:44.676874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.676781    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl" event={"ID":"b302c38f-92a7-4243-868b-3c3a04798313","Type":"ContainerDied","Data":"acc784ea146aaa70d158583b457a8fe15e243934948b567fb23c07c92b8c8c42"}
Oct 29 23:02:44.677331 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.677311    4559 scope.go:122] "RemoveContainer" containerID="acc784ea146aaa70d158583b457a8fe15e243934948b567fb23c07c92b8c8c42"
Oct 29 23:02:44.682929 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.682870    4559 scope.go:122] "RemoveContainer" containerID="2d4c5e1e7aad9fcb8afb2558b79b0fbb4fc09413189ba94f1f00493c36f85e59"
Oct 29 23:02:44.683166 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:02:44.683135    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d4c5e1e7aad9fcb8afb2558b79b0fbb4fc09413189ba94f1f00493c36f85e59\": container with ID starting with 2d4c5e1e7aad9fcb8afb2558b79b0fbb4fc09413189ba94f1f00493c36f85e59 not found: ID does not exist" containerID="2d4c5e1e7aad9fcb8afb2558b79b0fbb4fc09413189ba94f1f00493c36f85e59"
Oct 29 23:02:44.683307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.683176    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d4c5e1e7aad9fcb8afb2558b79b0fbb4fc09413189ba94f1f00493c36f85e59"} err="failed to get container status \"2d4c5e1e7aad9fcb8afb2558b79b0fbb4fc09413189ba94f1f00493c36f85e59\": rpc error: code = NotFound desc = could not find container \"2d4c5e1e7aad9fcb8afb2558b79b0fbb4fc09413189ba94f1f00493c36f85e59\": container with ID starting with 2d4c5e1e7aad9fcb8afb2558b79b0fbb4fc09413189ba94f1f00493c36f85e59 not found: ID does not exist"
Oct 29 23:02:44.683307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.683201    4559 scope.go:122] "RemoveContainer" containerID="c26327bd9adfcced2c9000b49138ad53851abebd5f3e711a1172975dea493efe"
Oct 29 23:02:44.683600 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.683532    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:02:44.683679 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.683619    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:02:44.683679 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.683672    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:02:44.787566 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.787536    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"]
Oct 29 23:02:44.791945 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.791913    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-eb02df-54d8869579-kckzx"]
Oct 29 23:02:44.836074 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.836039    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"]
Oct 29 23:02:44.852541 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:44.852373    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-eb02df-7b6dfb5df8-sw5mn"]
Oct 29 23:02:45.215477 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:45.215438    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:02:45.228671 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:45.228632    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="655d0083-da33-47ed-bb81-a3433a69da00" path="/var/lib/kubelet/pods/655d0083-da33-47ed-bb81-a3433a69da00/volumes"
Oct 29 23:02:45.229179 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:45.229164    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="700d065a-5535-41fb-bbc6-ba2fc54cd0a4" path="/var/lib/kubelet/pods/700d065a-5535-41fb-bbc6-ba2fc54cd0a4/volumes"
Oct 29 23:02:45.229572 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:45.229558    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c98b573a-0354-408a-a865-67705b3e070d" path="/var/lib/kubelet/pods/c98b573a-0354-408a-a865-67705b3e070d/volumes"
Oct 29 23:02:45.230269 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:45.230252    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5789ad1-f64c-4eb8-9a38-49da59afb179" path="/var/lib/kubelet/pods/d5789ad1-f64c-4eb8-9a38-49da59afb179/volumes"
Oct 29 23:02:45.230620 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:45.230606    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec5a59ee-c4bb-4457-8db1-9d098bd4147e" path="/var/lib/kubelet/pods/ec5a59ee-c4bb-4457-8db1-9d098bd4147e/volumes"
Oct 29 23:02:45.687449 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:45.687408    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl" event={"ID":"b302c38f-92a7-4243-868b-3c3a04798313","Type":"ContainerStarted","Data":"407159fd1464fd7c6323f9095cc550fa563acbd84836989fd22dcc5cf62354f7"}
Oct 29 23:02:46.103106 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:46.103010    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:02:46.104334 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:46.104304    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Oct 29 23:02:46.104476 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:46.104361    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl" podUID="b302c38f-92a7-4243-868b-3c3a04798313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Oct 29 23:02:47.103317 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.103279    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Oct 29 23:02:47.103744 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.103334    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl" podUID="b302c38f-92a7-4243-868b-3c3a04798313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Oct 29 23:02:47.522039 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.522006    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 29 23:02:47.522237 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.522065    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv" podUID="d9022f52-574d-478f-ae49-29732f126feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 29 23:02:47.713293 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.713247    4559 generic.go:356] "Generic (PLEG): container finished" podID="d9022f52-574d-478f-ae49-29732f126feb" containerID="9109a8eaded4ef240af04cd37dc62d3eb19ebd6c8978bc90290addbdcb385071" exitCode=0
Oct 29 23:02:47.713636 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.713503    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv" event={"ID":"d9022f52-574d-478f-ae49-29732f126feb","Type":"ContainerDied","Data":"9109a8eaded4ef240af04cd37dc62d3eb19ebd6c8978bc90290addbdcb385071"}
Oct 29 23:02:47.713636 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.713573    4559 scope.go:122] "RemoveContainer" containerID="9b08ddfa21728096ee900f21c3d2244423dc355eb363d052953db2589ea315a2"
Oct 29 23:02:47.863835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.858147    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 23:02:47.985558 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.985514    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9022f52-574d-478f-ae49-29732f126feb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9022f52-574d-478f-ae49-29732f126feb-istiod-ca-cert\") pod \"d9022f52-574d-478f-ae49-29732f126feb\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") "
Oct 29 23:02:47.985746 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.985639    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-istio-data\") pod \"d9022f52-574d-478f-ae49-29732f126feb\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") "
Oct 29 23:02:47.985862 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.985842    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-workload-certs\") pod \"d9022f52-574d-478f-ae49-29732f126feb\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") "
Oct 29 23:02:47.985922 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.985905    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-credential-socket\") pod \"d9022f52-574d-478f-ae49-29732f126feb\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") "
Oct 29 23:02:47.985999 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.985985    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9022f52-574d-478f-ae49-29732f126feb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9022f52-574d-478f-ae49-29732f126feb-istio-podinfo\") pod \"d9022f52-574d-478f-ae49-29732f126feb\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") "
Oct 29 23:02:47.986037 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.986017    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9022f52-574d-478f-ae49-29732f126feb-istio-token\" (UniqueName: \"kubernetes.io/projected/d9022f52-574d-478f-ae49-29732f126feb-istio-token\") pod \"d9022f52-574d-478f-ae49-29732f126feb\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") "
Oct 29 23:02:47.986072 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.986057    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-workload-socket\") pod \"d9022f52-574d-478f-ae49-29732f126feb\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") "
Oct 29 23:02:47.986111 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.986089    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-istio-envoy\") pod \"d9022f52-574d-478f-ae49-29732f126feb\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") "
Oct 29 23:02:47.986213 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.986197    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9022f52-574d-478f-ae49-29732f126feb-kube-api-access-k9wb7\" (UniqueName: \"kubernetes.io/projected/d9022f52-574d-478f-ae49-29732f126feb-kube-api-access-k9wb7\") pod \"d9022f52-574d-478f-ae49-29732f126feb\" (UID: \"d9022f52-574d-478f-ae49-29732f126feb\") "
Oct 29 23:02:47.987442 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.987420    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-credential-socket" pod "d9022f52-574d-478f-ae49-29732f126feb" (UID: "d9022f52-574d-478f-ae49-29732f126feb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:47.987698 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.987673    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9022f52-574d-478f-ae49-29732f126feb-istiod-ca-cert" pod "d9022f52-574d-478f-ae49-29732f126feb" (UID: "d9022f52-574d-478f-ae49-29732f126feb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:47.987882 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.987856    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-istio-data" pod "d9022f52-574d-478f-ae49-29732f126feb" (UID: "d9022f52-574d-478f-ae49-29732f126feb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:47.987882 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.987871    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-workload-socket" pod "d9022f52-574d-478f-ae49-29732f126feb" (UID: "d9022f52-574d-478f-ae49-29732f126feb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:47.988665 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.988463    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-workload-certs" pod "d9022f52-574d-478f-ae49-29732f126feb" (UID: "d9022f52-574d-478f-ae49-29732f126feb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:47.989981 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.989953    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9022f52-574d-478f-ae49-29732f126feb-kube-api-access-k9wb7" pod "d9022f52-574d-478f-ae49-29732f126feb" (UID: "d9022f52-574d-478f-ae49-29732f126feb"). InnerVolumeSpecName "kube-api-access-k9wb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:47.990410 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.990381    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9022f52-574d-478f-ae49-29732f126feb-istio-podinfo" pod "d9022f52-574d-478f-ae49-29732f126feb" (UID: "d9022f52-574d-478f-ae49-29732f126feb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:47.990660 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.990635    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-istio-envoy" pod "d9022f52-574d-478f-ae49-29732f126feb" (UID: "d9022f52-574d-478f-ae49-29732f126feb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:47.990790 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:47.990769    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9022f52-574d-478f-ae49-29732f126feb-istio-token" pod "d9022f52-574d-478f-ae49-29732f126feb" (UID: "d9022f52-574d-478f-ae49-29732f126feb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:48.088216 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:48.088125    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.088216 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:48.088158    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.088216 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:48.088170    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9022f52-574d-478f-ae49-29732f126feb-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.088216 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:48.088179    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9022f52-574d-478f-ae49-29732f126feb-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.088216 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:48.088188    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.088216 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:48.088195    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.088216 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:48.088203    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9wb7\" (UniqueName: \"kubernetes.io/projected/d9022f52-574d-478f-ae49-29732f126feb-kube-api-access-k9wb7\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.088216 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:48.088213    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9022f52-574d-478f-ae49-29732f126feb-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.088216 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:48.088222    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9022f52-574d-478f-ae49-29732f126feb-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.102611 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:48.102579    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Oct 29 23:02:48.102767 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:48.102649    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl" podUID="b302c38f-92a7-4243-868b-3c3a04798313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Oct 29 23:02:48.722644 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:48.722613    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"
Oct 29 23:02:48.723148 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:48.722640    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv" event={"ID":"d9022f52-574d-478f-ae49-29732f126feb","Type":"ContainerDied","Data":"97a03931317bd577c56446b73de6177e2fd0e22b657ccfd35f2a1ba87d31dd69"}
Oct 29 23:02:48.723148 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:48.722701    4559 scope.go:122] "RemoveContainer" containerID="9109a8eaded4ef240af04cd37dc62d3eb19ebd6c8978bc90290addbdcb385071"
Oct 29 23:02:48.792986 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:48.792907    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"]
Oct 29 23:02:48.804369 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:48.804332    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mshrv"]
Oct 29 23:02:49.065427 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:49.065338    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"]
Oct 29 23:02:49.102437 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:49.102396    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Oct 29 23:02:49.102651 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:49.102465    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl" podUID="b302c38f-92a7-4243-868b-3c3a04798313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Oct 29 23:02:49.232497 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:49.232461    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9022f52-574d-478f-ae49-29732f126feb" path="/var/lib/kubelet/pods/d9022f52-574d-478f-ae49-29732f126feb/volumes"
Oct 29 23:02:50.102980 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:50.102948    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Oct 29 23:02:50.103407 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:50.103002    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl" podUID="b302c38f-92a7-4243-868b-3c3a04798313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Oct 29 23:02:50.680245 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:50.678913    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"]
Oct 29 23:02:50.680767 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:50.680729    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm" podUID="74383eae-9fc4-44f4-9279-a2c085c4eeb9" containerName="istio-proxy" containerID="cri-o://62bf2da83f4d8881350b257564586f8198cd717364eed97a85a80567f5f8a6a7" gracePeriod=30
Oct 29 23:02:51.179800 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:51.179702    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:02:51.180491 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:51.180465    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:02:51.182036 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:51.182015    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:02:53.439862 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:53.439824    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"]
Oct 29 23:02:53.583755 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:53.583702    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765" podUID="fd73261d-a0be-4df4-99f6-73d2ea1c425a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:02:53.584387 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:53.583760    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62" podUID="290fd117-eaa7-4003-859d-3dd8ad075e05" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:02:54.066631 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:54.066587    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh" podUID="7776b504-8e2a-4138-b03a-60b47a61bd53" containerName="payload-processing" containerID="cri-o://3e30dfe662325dceb7fedc1bf7e7366c53ab76d86d12dbd752dc966a5245e554" gracePeriod=25
Oct 29 23:02:54.893849 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:54.890598    4559 generic.go:356] "Generic (PLEG): container finished" podID="fd73261d-a0be-4df4-99f6-73d2ea1c425a" containerID="b059f7beee73e4cdbf97ac042eacdeeb9a69324b82b9965eb6594230bf048b0b" exitCode=0
Oct 29 23:02:54.893849 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:54.890673    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765" event={"ID":"fd73261d-a0be-4df4-99f6-73d2ea1c425a","Type":"ContainerDied","Data":"b059f7beee73e4cdbf97ac042eacdeeb9a69324b82b9965eb6594230bf048b0b"}
Oct 29 23:02:54.893849 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:54.892903    4559 generic.go:356] "Generic (PLEG): container finished" podID="290fd117-eaa7-4003-859d-3dd8ad075e05" containerID="966dcdad4418935fdcb245ba3a4eea708534bd4b09a0e78bf9af2834381f8aa1" exitCode=0
Oct 29 23:02:54.893849 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:54.893017    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62" event={"ID":"290fd117-eaa7-4003-859d-3dd8ad075e05","Type":"ContainerDied","Data":"966dcdad4418935fdcb245ba3a4eea708534bd4b09a0e78bf9af2834381f8aa1"}
Oct 29 23:02:54.991769 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:54.991744    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"
Oct 29 23:02:54.996376 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:54.996352    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"
Oct 29 23:02:55.162085 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.162048    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd73261d-a0be-4df4-99f6-73d2ea1c425a-kube-api-access-262h8\" (UniqueName: \"kubernetes.io/projected/fd73261d-a0be-4df4-99f6-73d2ea1c425a-kube-api-access-262h8\") pod \"fd73261d-a0be-4df4-99f6-73d2ea1c425a\" (UID: \"fd73261d-a0be-4df4-99f6-73d2ea1c425a\") "
Oct 29 23:02:55.162299 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.162159    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290fd117-eaa7-4003-859d-3dd8ad075e05-kube-api-access-lkz8b\" (UniqueName: \"kubernetes.io/projected/290fd117-eaa7-4003-859d-3dd8ad075e05-kube-api-access-lkz8b\") pod \"290fd117-eaa7-4003-859d-3dd8ad075e05\" (UID: \"290fd117-eaa7-4003-859d-3dd8ad075e05\") "
Oct 29 23:02:55.162299 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.162206    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd73261d-a0be-4df4-99f6-73d2ea1c425a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd73261d-a0be-4df4-99f6-73d2ea1c425a-plugins-config-volume\") pod \"fd73261d-a0be-4df4-99f6-73d2ea1c425a\" (UID: \"fd73261d-a0be-4df4-99f6-73d2ea1c425a\") "
Oct 29 23:02:55.162299 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.162241    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/290fd117-eaa7-4003-859d-3dd8ad075e05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290fd117-eaa7-4003-859d-3dd8ad075e05-plugins-config-volume\") pod \"290fd117-eaa7-4003-859d-3dd8ad075e05\" (UID: \"290fd117-eaa7-4003-859d-3dd8ad075e05\") "
Oct 29 23:02:55.162746 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.162707    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290fd117-eaa7-4003-859d-3dd8ad075e05-plugins-config-volume" pod "290fd117-eaa7-4003-859d-3dd8ad075e05" (UID: "290fd117-eaa7-4003-859d-3dd8ad075e05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:55.162746 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.162718    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd73261d-a0be-4df4-99f6-73d2ea1c425a-plugins-config-volume" pod "fd73261d-a0be-4df4-99f6-73d2ea1c425a" (UID: "fd73261d-a0be-4df4-99f6-73d2ea1c425a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:55.164554 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.164530    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290fd117-eaa7-4003-859d-3dd8ad075e05-kube-api-access-lkz8b" pod "290fd117-eaa7-4003-859d-3dd8ad075e05" (UID: "290fd117-eaa7-4003-859d-3dd8ad075e05"). InnerVolumeSpecName "kube-api-access-lkz8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:55.164784 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.164768    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd73261d-a0be-4df4-99f6-73d2ea1c425a-kube-api-access-262h8" pod "fd73261d-a0be-4df4-99f6-73d2ea1c425a" (UID: "fd73261d-a0be-4df4-99f6-73d2ea1c425a"). InnerVolumeSpecName "kube-api-access-262h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:55.264595 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.264557    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-262h8\" (UniqueName: \"kubernetes.io/projected/fd73261d-a0be-4df4-99f6-73d2ea1c425a-kube-api-access-262h8\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.264595 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.264597    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkz8b\" (UniqueName: \"kubernetes.io/projected/290fd117-eaa7-4003-859d-3dd8ad075e05-kube-api-access-lkz8b\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.264821 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.264614    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd73261d-a0be-4df4-99f6-73d2ea1c425a-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.264821 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.264630    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290fd117-eaa7-4003-859d-3dd8ad075e05-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.903859 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.903833    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"
Oct 29 23:02:55.905918 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.903943    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62" event={"ID":"290fd117-eaa7-4003-859d-3dd8ad075e05","Type":"ContainerDied","Data":"49f1ca29624a7e39d75654ecb635c360779ff447843c10178a5201943eb6bd78"}
Oct 29 23:02:55.905918 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.904422    4559 scope.go:122] "RemoveContainer" containerID="966dcdad4418935fdcb245ba3a4eea708534bd4b09a0e78bf9af2834381f8aa1"
Oct 29 23:02:55.907534 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.907441    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765" event={"ID":"fd73261d-a0be-4df4-99f6-73d2ea1c425a","Type":"ContainerDied","Data":"3c6ea2a7ad01477bcb001210266b9113e034bb0593130a866a1c0e5f869d6b52"}
Oct 29 23:02:55.907702 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.907685    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"
Oct 29 23:02:55.916365 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.916335    4559 generic.go:356] "Generic (PLEG): container finished" podID="74383eae-9fc4-44f4-9279-a2c085c4eeb9" containerID="62bf2da83f4d8881350b257564586f8198cd717364eed97a85a80567f5f8a6a7" exitCode=0
Oct 29 23:02:55.916536 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.916406    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm" event={"ID":"74383eae-9fc4-44f4-9279-a2c085c4eeb9","Type":"ContainerDied","Data":"62bf2da83f4d8881350b257564586f8198cd717364eed97a85a80567f5f8a6a7"}
Oct 29 23:02:55.920413 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.920392    4559 scope.go:122] "RemoveContainer" containerID="b059f7beee73e4cdbf97ac042eacdeeb9a69324b82b9965eb6594230bf048b0b"
Oct 29 23:02:55.931867 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.931843    4559 scope.go:122] "RemoveContainer" containerID="c3a133d3083c54ae82895401079bdcdd5a7276abbfec38ddcccc7ff872dd37d1"
Oct 29 23:02:55.964662 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.964633    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"]
Oct 29 23:02:55.969722 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.969688    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-920032-77dc47dc96-xh765"]
Oct 29 23:02:55.994254 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.994219    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 23:02:55.996480 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:55.996452    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"]
Oct 29 23:02:56.001336 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.001313    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-920032-85fc75f8c5-fsg62"]
Oct 29 23:02:56.086926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.086887    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istiod-ca-cert\") pod \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") "
Oct 29 23:02:56.087132 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.086948    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-workload-certs\") pod \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") "
Oct 29 23:02:56.087132 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.087003    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-envoy\") pod \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") "
Oct 29 23:02:56.087132 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.087036    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-credential-socket\") pod \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") "
Oct 29 23:02:56.087132 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.087079    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-data\") pod \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") "
Oct 29 23:02:56.087132 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.087114    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-token\" (UniqueName: \"kubernetes.io/projected/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-token\") pod \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") "
Oct 29 23:02:56.087431 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.087155    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-workload-socket\") pod \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") "
Oct 29 23:02:56.087431 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.087324    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74383eae-9fc4-44f4-9279-a2c085c4eeb9-kube-api-access-b5btx\" (UniqueName: \"kubernetes.io/projected/74383eae-9fc4-44f4-9279-a2c085c4eeb9-kube-api-access-b5btx\") pod \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") "
Oct 29 23:02:56.087431 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.087391    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istiod-ca-cert" pod "74383eae-9fc4-44f4-9279-a2c085c4eeb9" (UID: "74383eae-9fc4-44f4-9279-a2c085c4eeb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:56.087431 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.087422    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-podinfo\") pod \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\" (UID: \"74383eae-9fc4-44f4-9279-a2c085c4eeb9\") "
Oct 29 23:02:56.087661 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.087574    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-workload-socket" pod "74383eae-9fc4-44f4-9279-a2c085c4eeb9" (UID: "74383eae-9fc4-44f4-9279-a2c085c4eeb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:56.088008 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.087978    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:56.088124 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.088011    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:56.089800 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.089767    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-credential-socket" pod "74383eae-9fc4-44f4-9279-a2c085c4eeb9" (UID: "74383eae-9fc4-44f4-9279-a2c085c4eeb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:56.090403 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.090355    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-data" pod "74383eae-9fc4-44f4-9279-a2c085c4eeb9" (UID: "74383eae-9fc4-44f4-9279-a2c085c4eeb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:56.090906 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.090787    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-envoy" pod "74383eae-9fc4-44f4-9279-a2c085c4eeb9" (UID: "74383eae-9fc4-44f4-9279-a2c085c4eeb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:56.091248 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.091225    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-workload-certs" pod "74383eae-9fc4-44f4-9279-a2c085c4eeb9" (UID: "74383eae-9fc4-44f4-9279-a2c085c4eeb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:56.092563 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.092481    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-podinfo" pod "74383eae-9fc4-44f4-9279-a2c085c4eeb9" (UID: "74383eae-9fc4-44f4-9279-a2c085c4eeb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:56.092670 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.092570    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-token" pod "74383eae-9fc4-44f4-9279-a2c085c4eeb9" (UID: "74383eae-9fc4-44f4-9279-a2c085c4eeb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:56.092996 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.092970    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74383eae-9fc4-44f4-9279-a2c085c4eeb9-kube-api-access-b5btx" pod "74383eae-9fc4-44f4-9279-a2c085c4eeb9" (UID: "74383eae-9fc4-44f4-9279-a2c085c4eeb9"). InnerVolumeSpecName "kube-api-access-b5btx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:56.189285 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.189244    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5btx\" (UniqueName: \"kubernetes.io/projected/74383eae-9fc4-44f4-9279-a2c085c4eeb9-kube-api-access-b5btx\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:56.189285 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.189275    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:56.189285 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.189287    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:56.189285 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.189294    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:56.189569 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.189303    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:56.189569 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.189311    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:56.189569 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.189319    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74383eae-9fc4-44f4-9279-a2c085c4eeb9-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:02:56.561370 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.561324    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"]
Oct 29 23:02:56.561648 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.561620    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2" podUID="29b7beb6-4444-4752-9f17-c5b76b8eb2f9" containerName="istio-proxy" containerID="cri-o://72d8e52e3e5ec1583ab3e989aa03f825c40095b772eca7904d4c1618dbebe1e4" gracePeriod=30
Oct 29 23:02:56.825891 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.824886    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"]
Oct 29 23:02:56.825891 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.825856    4559 cpu_manager.go:414] "containerMap: removing container" podUID="655d0083-da33-47ed-bb81-a3433a69da00" containerName="payload-pre-processing"
Oct 29 23:02:56.825891 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.825876    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="655d0083-da33-47ed-bb81-a3433a69da00" containerName="payload-pre-processing"
Oct 29 23:02:56.825891 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.825893    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ec5a59ee-c4bb-4457-8db1-9d098bd4147e" containerName="payload-processing"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.825901    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec5a59ee-c4bb-4457-8db1-9d098bd4147e" containerName="payload-processing"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.825919    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d5789ad1-f64c-4eb8-9a38-49da59afb179" containerName="payload-processing"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.825927    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5789ad1-f64c-4eb8-9a38-49da59afb179" containerName="payload-processing"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.825950    4559 cpu_manager.go:414] "containerMap: removing container" podUID="700d065a-5535-41fb-bbc6-ba2fc54cd0a4" containerName="istio-proxy"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.825958    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700d065a-5535-41fb-bbc6-ba2fc54cd0a4" containerName="istio-proxy"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.825976    4559 cpu_manager.go:414] "containerMap: removing container" podUID="290fd117-eaa7-4003-859d-3dd8ad075e05" containerName="payload-pre-processing"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.825983    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290fd117-eaa7-4003-859d-3dd8ad075e05" containerName="payload-pre-processing"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826005    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d8f6c0c6-84c4-424e-a740-9e2dd5fe332c" containerName="revoke-keys"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826012    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f6c0c6-84c4-424e-a740-9e2dd5fe332c" containerName="revoke-keys"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826022    4559 cpu_manager.go:414] "containerMap: removing container" podUID="fd73261d-a0be-4df4-99f6-73d2ea1c425a" containerName="payload-processing"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826028    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd73261d-a0be-4df4-99f6-73d2ea1c425a" containerName="payload-processing"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826038    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d9022f52-574d-478f-ae49-29732f126feb" containerName="istio-proxy"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826045    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9022f52-574d-478f-ae49-29732f126feb" containerName="istio-proxy"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826080    4559 cpu_manager.go:414] "containerMap: removing container" podUID="74383eae-9fc4-44f4-9279-a2c085c4eeb9" containerName="istio-proxy"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826088    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74383eae-9fc4-44f4-9279-a2c085c4eeb9" containerName="istio-proxy"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826102    4559 cpu_manager.go:414] "containerMap: removing container" podUID="74383eae-9fc4-44f4-9279-a2c085c4eeb9" containerName="istio-proxy"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826108    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74383eae-9fc4-44f4-9279-a2c085c4eeb9" containerName="istio-proxy"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826135    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d9022f52-574d-478f-ae49-29732f126feb" containerName="istio-proxy"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826142    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9022f52-574d-478f-ae49-29732f126feb" containerName="istio-proxy"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826290    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="700d065a-5535-41fb-bbc6-ba2fc54cd0a4" containerName="istio-proxy"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826302    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9022f52-574d-478f-ae49-29732f126feb" containerName="istio-proxy"
Oct 29 23:02:56.826307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826313    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="655d0083-da33-47ed-bb81-a3433a69da00" containerName="payload-pre-processing"
Oct 29 23:02:56.827769 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826324    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec5a59ee-c4bb-4457-8db1-9d098bd4147e" containerName="payload-processing"
Oct 29 23:02:56.827769 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826333    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5789ad1-f64c-4eb8-9a38-49da59afb179" containerName="payload-processing"
Oct 29 23:02:56.827769 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826344    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="290fd117-eaa7-4003-859d-3dd8ad075e05" containerName="payload-pre-processing"
Oct 29 23:02:56.827769 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826356    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f6c0c6-84c4-424e-a740-9e2dd5fe332c" containerName="revoke-keys"
Oct 29 23:02:56.827769 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826365    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd73261d-a0be-4df4-99f6-73d2ea1c425a" containerName="payload-processing"
Oct 29 23:02:56.827769 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826377    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="74383eae-9fc4-44f4-9279-a2c085c4eeb9" containerName="istio-proxy"
Oct 29 23:02:56.827769 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.826385    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="74383eae-9fc4-44f4-9279-a2c085c4eeb9" containerName="istio-proxy"
Oct 29 23:02:56.834425 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.832727    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"
Oct 29 23:02:56.865417 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.849276    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9d6d4e\""
Oct 29 23:02:56.865417 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.853096    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9d6d4e-dockercfg-kqmq4\""
Oct 29 23:02:56.867941 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.867897    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"]
Oct 29 23:02:56.931855 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.930505    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"
Oct 29 23:02:56.931855 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.930537    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm" event={"ID":"74383eae-9fc4-44f4-9279-a2c085c4eeb9","Type":"ContainerDied","Data":"edffb347c075fab401bdd3286a3aa6b74e702bbbf7dfc5dbf8f538fbcd0db55a"}
Oct 29 23:02:56.931855 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.930610    4559 scope.go:122] "RemoveContainer" containerID="62bf2da83f4d8881350b257564586f8198cd717364eed97a85a80567f5f8a6a7"
Oct 29 23:02:56.984573 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.984529    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"]
Oct 29 23:02:56.990566 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:56.990522    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-85862e-openshift-default-7b794c9f49-6wxwm"]
Oct 29 23:02:57.000593 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.000536    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b5mf\" (UniqueName: \"kubernetes.io/projected/e367910b-b67d-499e-a1dd-3eda32d1fa33-kube-api-access-5b5mf\") pod \"payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw\" (UID: \"e367910b-b67d-499e-a1dd-3eda32d1fa33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"
Oct 29 23:02:57.000802 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.000687    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e367910b-b67d-499e-a1dd-3eda32d1fa33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw\" (UID: \"e367910b-b67d-499e-a1dd-3eda32d1fa33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"
Oct 29 23:02:57.001803 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.001777    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"]
Oct 29 23:02:57.002838 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.002797    4559 cpu_manager.go:414] "containerMap: removing container" podUID="700d065a-5535-41fb-bbc6-ba2fc54cd0a4" containerName="istio-proxy"
Oct 29 23:02:57.002892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.002843    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700d065a-5535-41fb-bbc6-ba2fc54cd0a4" containerName="istio-proxy"
Oct 29 23:02:57.003060 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.003046    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="700d065a-5535-41fb-bbc6-ba2fc54cd0a4" containerName="istio-proxy"
Oct 29 23:02:57.003097 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.003067    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9022f52-574d-478f-ae49-29732f126feb" containerName="istio-proxy"
Oct 29 23:02:57.008825 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.008785    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"
Oct 29 23:02:57.044881 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.025787    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"]
Oct 29 23:02:57.103079 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.102985    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303751f6-3328-401c-9f2c-4970a54ae09c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q\" (UID: \"303751f6-3328-401c-9f2c-4970a54ae09c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"
Oct 29 23:02:57.103079 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.103034    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dmmf\" (UniqueName: \"kubernetes.io/projected/303751f6-3328-401c-9f2c-4970a54ae09c-kube-api-access-9dmmf\") pod \"payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q\" (UID: \"303751f6-3328-401c-9f2c-4970a54ae09c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"
Oct 29 23:02:57.103288 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.103139    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b5mf\" (UniqueName: \"kubernetes.io/projected/e367910b-b67d-499e-a1dd-3eda32d1fa33-kube-api-access-5b5mf\") pod \"payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw\" (UID: \"e367910b-b67d-499e-a1dd-3eda32d1fa33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"
Oct 29 23:02:57.103288 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.103178    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e367910b-b67d-499e-a1dd-3eda32d1fa33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw\" (UID: \"e367910b-b67d-499e-a1dd-3eda32d1fa33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"
Oct 29 23:02:57.103948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.103924    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e367910b-b67d-499e-a1dd-3eda32d1fa33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw\" (UID: \"e367910b-b67d-499e-a1dd-3eda32d1fa33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"
Oct 29 23:02:57.124285 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.124246    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b5mf\" (UniqueName: \"kubernetes.io/projected/e367910b-b67d-499e-a1dd-3eda32d1fa33-kube-api-access-5b5mf\") pod \"payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw\" (UID: \"e367910b-b67d-499e-a1dd-3eda32d1fa33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"
Oct 29 23:02:57.202260 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.202221    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"
Oct 29 23:02:57.204210 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.204172    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303751f6-3328-401c-9f2c-4970a54ae09c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q\" (UID: \"303751f6-3328-401c-9f2c-4970a54ae09c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"
Oct 29 23:02:57.204381 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.204235    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dmmf\" (UniqueName: \"kubernetes.io/projected/303751f6-3328-401c-9f2c-4970a54ae09c-kube-api-access-9dmmf\") pod \"payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q\" (UID: \"303751f6-3328-401c-9f2c-4970a54ae09c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"
Oct 29 23:02:57.204973 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.204952    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303751f6-3328-401c-9f2c-4970a54ae09c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q\" (UID: \"303751f6-3328-401c-9f2c-4970a54ae09c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"
Oct 29 23:02:57.224938 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.224907    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dmmf\" (UniqueName: \"kubernetes.io/projected/303751f6-3328-401c-9f2c-4970a54ae09c-kube-api-access-9dmmf\") pod \"payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q\" (UID: \"303751f6-3328-401c-9f2c-4970a54ae09c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"
Oct 29 23:02:57.229763 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.229726    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="290fd117-eaa7-4003-859d-3dd8ad075e05" path="/var/lib/kubelet/pods/290fd117-eaa7-4003-859d-3dd8ad075e05/volumes"
Oct 29 23:02:57.230371 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.230354    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74383eae-9fc4-44f4-9279-a2c085c4eeb9" path="/var/lib/kubelet/pods/74383eae-9fc4-44f4-9279-a2c085c4eeb9/volumes"
Oct 29 23:02:57.230750 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.230737    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd73261d-a0be-4df4-99f6-73d2ea1c425a" path="/var/lib/kubelet/pods/fd73261d-a0be-4df4-99f6-73d2ea1c425a/volumes"
Oct 29 23:02:57.361657 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.361549    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"
Oct 29 23:02:57.540919 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.540881    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"]
Oct 29 23:02:57.543082 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:02:57.543040    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode367910b_b67d_499e_a1dd_3eda32d1fa33.slice/crio-33e70e7280adfeb943969ceffe263260d5c48cb6499a82c1cf35573771cac946 WatchSource:0}: Error finding container 33e70e7280adfeb943969ceffe263260d5c48cb6499a82c1cf35573771cac946: Status 404 returned error can't find the container with id 33e70e7280adfeb943969ceffe263260d5c48cb6499a82c1cf35573771cac946
Oct 29 23:02:57.818387 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.818355    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"]
Oct 29 23:02:57.819859 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:02:57.819802    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303751f6_3328_401c_9f2c_4970a54ae09c.slice/crio-4e3bde41c9ef565959d5afb1de3212b6320944a3f16dd022fd1ef5fb3690b986 WatchSource:0}: Error finding container 4e3bde41c9ef565959d5afb1de3212b6320944a3f16dd022fd1ef5fb3690b986: Status 404 returned error can't find the container with id 4e3bde41c9ef565959d5afb1de3212b6320944a3f16dd022fd1ef5fb3690b986
Oct 29 23:02:57.972160 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.970795    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw" event={"ID":"e367910b-b67d-499e-a1dd-3eda32d1fa33","Type":"ContainerStarted","Data":"d7c36dad33f834ab05258be2df62782b7e914f3c095fdb128b1e48ec9179d7db"}
Oct 29 23:02:57.972160 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.970897    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw" event={"ID":"e367910b-b67d-499e-a1dd-3eda32d1fa33","Type":"ContainerStarted","Data":"33e70e7280adfeb943969ceffe263260d5c48cb6499a82c1cf35573771cac946"}
Oct 29 23:02:57.972160 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.972091    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"
Oct 29 23:02:57.997129 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.997090    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q" event={"ID":"303751f6-3328-401c-9f2c-4970a54ae09c","Type":"ContainerStarted","Data":"33e8a0701e5057ce81e167e2ec82974736ba087eb315e27b7d5042443f21b7eb"}
Oct 29 23:02:57.997395 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.997380    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q" event={"ID":"303751f6-3328-401c-9f2c-4970a54ae09c","Type":"ContainerStarted","Data":"4e3bde41c9ef565959d5afb1de3212b6320944a3f16dd022fd1ef5fb3690b986"}
Oct 29 23:02:57.998527 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:57.998283    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"
Oct 29 23:02:58.020821 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:58.020752    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw" podStartSLOduration=2.020732465 podStartE2EDuration="2.020732465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:58.017530607 +0000 UTC m=+2916.928974163" watchObservedRunningTime="2026-10-29 23:02:58.020732465 +0000 UTC m=+2916.932176021"
Oct 29 23:02:58.441000 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:58.440948    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq" podUID="d190385b-94ea-4b39-9318-3a33e14e8286" containerName="payload-processing" containerID="cri-o://6f11f03f3179b369d195ccfa03394f3f21a5e21315a56272e5d455f1abf2a78d" gracePeriod=25
Oct 29 23:02:58.891697 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:58.891637    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q" podStartSLOduration=2.891615001 podStartE2EDuration="2.891615001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:58.056994529 +0000 UTC m=+2916.968438083" watchObservedRunningTime="2026-10-29 23:02:58.891615001 +0000 UTC m=+2917.803058555"
Oct 29 23:02:58.893784 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:02:58.893759    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:03:01.645532 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.645493    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"]
Oct 29 23:03:01.680538 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.680503    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"]
Oct 29 23:03:01.798081 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.798039    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh" podUID="7776b504-8e2a-4138-b03a-60b47a61bd53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:01.921325 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.919288    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 23:03:01.987795 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.987765    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-podinfo\") pod \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") "
Oct 29 23:03:01.988012 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.987830    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istiod-ca-cert\") pod \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") "
Oct 29 23:03:01.988012 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.987862    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-workload-certs\") pod \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") "
Oct 29 23:03:01.988012 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.987890    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-workload-socket\") pod \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") "
Oct 29 23:03:01.988012 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.987957    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-envoy\") pod \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") "
Oct 29 23:03:01.988012 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.987992    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-kube-api-access-7gmtk\" (UniqueName: \"kubernetes.io/projected/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-kube-api-access-7gmtk\") pod \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") "
Oct 29 23:03:01.988312 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.988096    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-data\") pod \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") "
Oct 29 23:03:01.988312 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.988149    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-token\" (UniqueName: \"kubernetes.io/projected/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-token\") pod \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") "
Oct 29 23:03:01.988312 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.988256    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istiod-ca-cert" pod "29b7beb6-4444-4752-9f17-c5b76b8eb2f9" (UID: "29b7beb6-4444-4752-9f17-c5b76b8eb2f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:01.988486 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.988373    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-workload-socket" pod "29b7beb6-4444-4752-9f17-c5b76b8eb2f9" (UID: "29b7beb6-4444-4752-9f17-c5b76b8eb2f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:01.988550 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.988517    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-data" pod "29b7beb6-4444-4752-9f17-c5b76b8eb2f9" (UID: "29b7beb6-4444-4752-9f17-c5b76b8eb2f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:01.988616 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.988543    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-workload-certs" pod "29b7beb6-4444-4752-9f17-c5b76b8eb2f9" (UID: "29b7beb6-4444-4752-9f17-c5b76b8eb2f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:01.988678 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.988620    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-credential-socket\") pod \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\" (UID: \"29b7beb6-4444-4752-9f17-c5b76b8eb2f9\") "
Oct 29 23:03:01.988948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.988919    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-credential-socket" pod "29b7beb6-4444-4752-9f17-c5b76b8eb2f9" (UID: "29b7beb6-4444-4752-9f17-c5b76b8eb2f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:01.989331 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.989312    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:01.989392 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.989339    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:01.989392 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.989355    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:01.989392 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.989368    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:01.989392 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.989381    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:01.990849 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.990824    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-token" pod "29b7beb6-4444-4752-9f17-c5b76b8eb2f9" (UID: "29b7beb6-4444-4752-9f17-c5b76b8eb2f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:01.991850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.991788    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-kube-api-access-7gmtk" pod "29b7beb6-4444-4752-9f17-c5b76b8eb2f9" (UID: "29b7beb6-4444-4752-9f17-c5b76b8eb2f9"). InnerVolumeSpecName "kube-api-access-7gmtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:01.994215 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.994154    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-podinfo" pod "29b7beb6-4444-4752-9f17-c5b76b8eb2f9" (UID: "29b7beb6-4444-4752-9f17-c5b76b8eb2f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:01.995454 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:01.995426    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-envoy" pod "29b7beb6-4444-4752-9f17-c5b76b8eb2f9" (UID: "29b7beb6-4444-4752-9f17-c5b76b8eb2f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:02.040366 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:02.040332    4559 generic.go:356] "Generic (PLEG): container finished" podID="29b7beb6-4444-4752-9f17-c5b76b8eb2f9" containerID="72d8e52e3e5ec1583ab3e989aa03f825c40095b772eca7904d4c1618dbebe1e4" exitCode=0
Oct 29 23:03:02.040623 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:02.040392    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"
Oct 29 23:03:02.041360 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:02.040383    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2" event={"ID":"29b7beb6-4444-4752-9f17-c5b76b8eb2f9","Type":"ContainerDied","Data":"72d8e52e3e5ec1583ab3e989aa03f825c40095b772eca7904d4c1618dbebe1e4"}
Oct 29 23:03:02.041360 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:02.040738    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2" event={"ID":"29b7beb6-4444-4752-9f17-c5b76b8eb2f9","Type":"ContainerDied","Data":"cce62d1b396439c6b4830aa549e29054af27e26c304884a3b812853051ebb784"}
Oct 29 23:03:02.041360 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:02.040771    4559 scope.go:122] "RemoveContainer" containerID="72d8e52e3e5ec1583ab3e989aa03f825c40095b772eca7904d4c1618dbebe1e4"
Oct 29 23:03:02.057465 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:02.057441    4559 scope.go:122] "RemoveContainer" containerID="72d8e52e3e5ec1583ab3e989aa03f825c40095b772eca7904d4c1618dbebe1e4"
Oct 29 23:03:02.057895 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:02.057868    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d8e52e3e5ec1583ab3e989aa03f825c40095b772eca7904d4c1618dbebe1e4\": container with ID starting with 72d8e52e3e5ec1583ab3e989aa03f825c40095b772eca7904d4c1618dbebe1e4 not found: ID does not exist" containerID="72d8e52e3e5ec1583ab3e989aa03f825c40095b772eca7904d4c1618dbebe1e4"
Oct 29 23:03:02.057995 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:02.057905    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d8e52e3e5ec1583ab3e989aa03f825c40095b772eca7904d4c1618dbebe1e4"} err="failed to get container status \"72d8e52e3e5ec1583ab3e989aa03f825c40095b772eca7904d4c1618dbebe1e4\": rpc error: code = NotFound desc = could not find container \"72d8e52e3e5ec1583ab3e989aa03f825c40095b772eca7904d4c1618dbebe1e4\": container with ID starting with 72d8e52e3e5ec1583ab3e989aa03f825c40095b772eca7904d4c1618dbebe1e4 not found: ID does not exist"
Oct 29 23:03:02.090261 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:02.090220    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:02.090261 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:02.090255    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gmtk\" (UniqueName: \"kubernetes.io/projected/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-kube-api-access-7gmtk\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:02.090484 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:02.090271    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:02.090484 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:02.090283    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b7beb6-4444-4752-9f17-c5b76b8eb2f9-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:02.097402 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:02.097367    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"]
Oct 29 23:03:02.103676 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:02.103647    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-ht9j2"]
Oct 29 23:03:03.124533 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:03.124450    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq" podUID="d190385b-94ea-4b39-9318-3a33e14e8286" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:03.228194 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:03.228156    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b7beb6-4444-4752-9f17-c5b76b8eb2f9" path="/var/lib/kubelet/pods/29b7beb6-4444-4752-9f17-c5b76b8eb2f9/volumes"
Oct 29 23:03:04.747879 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:04.747838    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"]
Oct 29 23:03:04.748411 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:04.748121    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs" podUID="e09d2003-0997-418a-8fa3-f64c92d28572" containerName="istio-proxy" containerID="cri-o://8397c838f1cde156413c7e241b28be7a820f89ff01bf91ca9d5edf39135866f1" gracePeriod=30
Oct 29 23:03:06.646257 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:06.646206    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h" podUID="299a03af-cb40-46b1-b42d-8afa20c7cd00" containerName="payload-processing" containerID="cri-o://f27e188cea94f371d194cfe0a0111d1ef6f7a5f44fa93061767d9fefd38867ae" gracePeriod=25
Oct 29 23:03:06.681518 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:06.681469    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr" podUID="341bcfef-7b8c-4732-a60c-2849614df3b7" containerName="payload-pre-processing" containerID="cri-o://86e6ce7b87b79adf712b19c0b2f6885ef9b5cf53e0028e85423f5ea71ac8ed15" gracePeriod=25
Oct 29 23:03:10.020960 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.020867    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"
Oct 29 23:03:10.021696 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.021673    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"
Oct 29 23:03:10.027738 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.027711    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 23:03:10.097595 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.097540    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e09d2003-0997-418a-8fa3-f64c92d28572-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09d2003-0997-418a-8fa3-f64c92d28572-istio-podinfo\") pod \"e09d2003-0997-418a-8fa3-f64c92d28572\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") "
Oct 29 23:03:10.097802 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.097613    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09d2003-0997-418a-8fa3-f64c92d28572-istio-token\" (UniqueName: \"kubernetes.io/projected/e09d2003-0997-418a-8fa3-f64c92d28572-istio-token\") pod \"e09d2003-0997-418a-8fa3-f64c92d28572\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") "
Oct 29 23:03:10.097897 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.097855    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-workload-certs\") pod \"e09d2003-0997-418a-8fa3-f64c92d28572\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") "
Oct 29 23:03:10.097950 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.097923    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e09d2003-0997-418a-8fa3-f64c92d28572-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09d2003-0997-418a-8fa3-f64c92d28572-istiod-ca-cert\") pod \"e09d2003-0997-418a-8fa3-f64c92d28572\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") "
Oct 29 23:03:10.098007 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.097979    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-istio-data\") pod \"e09d2003-0997-418a-8fa3-f64c92d28572\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") "
Oct 29 23:03:10.098063 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.098010    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09d2003-0997-418a-8fa3-f64c92d28572-kube-api-access-sg5sg\" (UniqueName: \"kubernetes.io/projected/e09d2003-0997-418a-8fa3-f64c92d28572-kube-api-access-sg5sg\") pod \"e09d2003-0997-418a-8fa3-f64c92d28572\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") "
Oct 29 23:03:10.098189 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.098167    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-workload-socket\") pod \"e09d2003-0997-418a-8fa3-f64c92d28572\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") "
Oct 29 23:03:10.098951 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.098926    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-credential-socket\") pod \"e09d2003-0997-418a-8fa3-f64c92d28572\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") "
Oct 29 23:03:10.099204 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.098309    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-workload-certs" pod "e09d2003-0997-418a-8fa3-f64c92d28572" (UID: "e09d2003-0997-418a-8fa3-f64c92d28572"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:10.099204 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.098663    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-workload-socket" pod "e09d2003-0997-418a-8fa3-f64c92d28572" (UID: "e09d2003-0997-418a-8fa3-f64c92d28572"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:10.099204 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.098735    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-istio-data" pod "e09d2003-0997-418a-8fa3-f64c92d28572" (UID: "e09d2003-0997-418a-8fa3-f64c92d28572"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:10.099432 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.099228    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-credential-socket" pod "e09d2003-0997-418a-8fa3-f64c92d28572" (UID: "e09d2003-0997-418a-8fa3-f64c92d28572"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:10.099649 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.099615    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09d2003-0997-418a-8fa3-f64c92d28572-istiod-ca-cert" pod "e09d2003-0997-418a-8fa3-f64c92d28572" (UID: "e09d2003-0997-418a-8fa3-f64c92d28572"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:10.100960 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.100925    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09d2003-0997-418a-8fa3-f64c92d28572-istio-token" pod "e09d2003-0997-418a-8fa3-f64c92d28572" (UID: "e09d2003-0997-418a-8fa3-f64c92d28572"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:10.101852 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.101783    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e09d2003-0997-418a-8fa3-f64c92d28572-istio-podinfo" pod "e09d2003-0997-418a-8fa3-f64c92d28572" (UID: "e09d2003-0997-418a-8fa3-f64c92d28572"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:10.102300 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.102280    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09d2003-0997-418a-8fa3-f64c92d28572-kube-api-access-sg5sg" pod "e09d2003-0997-418a-8fa3-f64c92d28572" (UID: "e09d2003-0997-418a-8fa3-f64c92d28572"). InnerVolumeSpecName "kube-api-access-sg5sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:10.102949 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.102925    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-istio-envoy\") pod \"e09d2003-0997-418a-8fa3-f64c92d28572\" (UID: \"e09d2003-0997-418a-8fa3-f64c92d28572\") "
Oct 29 23:03:10.104731 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.104602    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.104731 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.104629    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.104731 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.104645    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09d2003-0997-418a-8fa3-f64c92d28572-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.104731 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.104659    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09d2003-0997-418a-8fa3-f64c92d28572-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.104731 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.104671    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.104731 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.104683    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09d2003-0997-418a-8fa3-f64c92d28572-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.104731 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.104695    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.104731 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.104708    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg5sg\" (UniqueName: \"kubernetes.io/projected/e09d2003-0997-418a-8fa3-f64c92d28572-kube-api-access-sg5sg\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.106522 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.106460    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-istio-envoy" pod "e09d2003-0997-418a-8fa3-f64c92d28572" (UID: "e09d2003-0997-418a-8fa3-f64c92d28572"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:10.144105 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.144065    4559 generic.go:356] "Generic (PLEG): container finished" podID="e09d2003-0997-418a-8fa3-f64c92d28572" containerID="8397c838f1cde156413c7e241b28be7a820f89ff01bf91ca9d5edf39135866f1" exitCode=0
Oct 29 23:03:10.144276 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.144137    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"
Oct 29 23:03:10.144276 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.144158    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs" event={"ID":"e09d2003-0997-418a-8fa3-f64c92d28572","Type":"ContainerDied","Data":"8397c838f1cde156413c7e241b28be7a820f89ff01bf91ca9d5edf39135866f1"}
Oct 29 23:03:10.144276 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.144238    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs" event={"ID":"e09d2003-0997-418a-8fa3-f64c92d28572","Type":"ContainerDied","Data":"ab9150ff73f98b72bb7561ba185bac703df855158b92d65d5a85839383a8efaf"}
Oct 29 23:03:10.144276 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.144269    4559 scope.go:122] "RemoveContainer" containerID="8397c838f1cde156413c7e241b28be7a820f89ff01bf91ca9d5edf39135866f1"
Oct 29 23:03:10.161516 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.161426    4559 scope.go:122] "RemoveContainer" containerID="01e3d92ce7e5890894e7e1f66c4380832ad287ef5508844f84b3c986e0ca9472"
Oct 29 23:03:10.178181 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.177831    4559 scope.go:122] "RemoveContainer" containerID="8397c838f1cde156413c7e241b28be7a820f89ff01bf91ca9d5edf39135866f1"
Oct 29 23:03:10.178457 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:10.178424    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8397c838f1cde156413c7e241b28be7a820f89ff01bf91ca9d5edf39135866f1\": container with ID starting with 8397c838f1cde156413c7e241b28be7a820f89ff01bf91ca9d5edf39135866f1 not found: ID does not exist" containerID="8397c838f1cde156413c7e241b28be7a820f89ff01bf91ca9d5edf39135866f1"
Oct 29 23:03:10.178568 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.178468    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8397c838f1cde156413c7e241b28be7a820f89ff01bf91ca9d5edf39135866f1"} err="failed to get container status \"8397c838f1cde156413c7e241b28be7a820f89ff01bf91ca9d5edf39135866f1\": rpc error: code = NotFound desc = could not find container \"8397c838f1cde156413c7e241b28be7a820f89ff01bf91ca9d5edf39135866f1\": container with ID starting with 8397c838f1cde156413c7e241b28be7a820f89ff01bf91ca9d5edf39135866f1 not found: ID does not exist"
Oct 29 23:03:10.178568 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.178495    4559 scope.go:122] "RemoveContainer" containerID="01e3d92ce7e5890894e7e1f66c4380832ad287ef5508844f84b3c986e0ca9472"
Oct 29 23:03:10.178847 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:10.178794    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01e3d92ce7e5890894e7e1f66c4380832ad287ef5508844f84b3c986e0ca9472\": container with ID starting with 01e3d92ce7e5890894e7e1f66c4380832ad287ef5508844f84b3c986e0ca9472 not found: ID does not exist" containerID="01e3d92ce7e5890894e7e1f66c4380832ad287ef5508844f84b3c986e0ca9472"
Oct 29 23:03:10.178931 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.178846    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01e3d92ce7e5890894e7e1f66c4380832ad287ef5508844f84b3c986e0ca9472"} err="failed to get container status \"01e3d92ce7e5890894e7e1f66c4380832ad287ef5508844f84b3c986e0ca9472\": rpc error: code = NotFound desc = could not find container \"01e3d92ce7e5890894e7e1f66c4380832ad287ef5508844f84b3c986e0ca9472\": container with ID starting with 01e3d92ce7e5890894e7e1f66c4380832ad287ef5508844f84b3c986e0ca9472 not found: ID does not exist"
Oct 29 23:03:10.230241 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.230197    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"]
Oct 29 23:03:10.232093 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.232062    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09d2003-0997-418a-8fa3-f64c92d28572-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.236933 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.236905    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"]
Oct 29 23:03:10.242282 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:10.242247    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-kwffs"]
Oct 29 23:03:11.235099 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:11.235057    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e09d2003-0997-418a-8fa3-f64c92d28572" path="/var/lib/kubelet/pods/e09d2003-0997-418a-8fa3-f64c92d28572/volumes"
Oct 29 23:03:11.796136 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:11.796089    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh" podUID="7776b504-8e2a-4138-b03a-60b47a61bd53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:12.482220 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:12.482186    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h" podUID="299a03af-cb40-46b1-b42d-8afa20c7cd00" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:13.124073 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:13.124030    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq" podUID="d190385b-94ea-4b39-9318-3a33e14e8286" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:13.695697 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:13.695658    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:03:13.817847 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:13.817800    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"]
Oct 29 23:03:13.818152 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:13.818025    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2" podUID="a1040d20-45e0-4df0-bc5a-2ca67e5e5108" containerName="maas-api" containerID="cri-o://6ca7c57a1b042aebfaae4b0fc22e934591411b859f3c7ae4b4e97956bc7237bd" gracePeriod=30
Oct 29 23:03:13.849345 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:13.849305    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2" podUID="a1040d20-45e0-4df0-bc5a-2ca67e5e5108" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.152:8443/health\": dial tcp 10.134.0.152:8443: connect: connection refused"
Oct 29 23:03:14.242259 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.230423    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 23:03:14.277893 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.277460    4559 generic.go:356] "Generic (PLEG): container finished" podID="7776b504-8e2a-4138-b03a-60b47a61bd53" containerID="3e30dfe662325dceb7fedc1bf7e7366c53ab76d86d12dbd752dc966a5245e554" exitCode=0
Oct 29 23:03:14.277893 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.277720    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh" event={"ID":"7776b504-8e2a-4138-b03a-60b47a61bd53","Type":"ContainerDied","Data":"3e30dfe662325dceb7fedc1bf7e7366c53ab76d86d12dbd752dc966a5245e554"}
Oct 29 23:03:14.295348 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.295309    4559 generic.go:356] "Generic (PLEG): container finished" podID="a1040d20-45e0-4df0-bc5a-2ca67e5e5108" containerID="6ca7c57a1b042aebfaae4b0fc22e934591411b859f3c7ae4b4e97956bc7237bd" exitCode=0
Oct 29 23:03:14.295569 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.295419    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2" event={"ID":"a1040d20-45e0-4df0-bc5a-2ca67e5e5108","Type":"ContainerDied","Data":"6ca7c57a1b042aebfaae4b0fc22e934591411b859f3c7ae4b4e97956bc7237bd"}
Oct 29 23:03:14.295569 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.295469    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2" event={"ID":"a1040d20-45e0-4df0-bc5a-2ca67e5e5108","Type":"ContainerDied","Data":"a971c384e33ae22ce45bde55c6647a9365d838469cb81b295e0ee921a058a2b7"}
Oct 29 23:03:14.295569 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.295499    4559 scope.go:122] "RemoveContainer" containerID="6ca7c57a1b042aebfaae4b0fc22e934591411b859f3c7ae4b4e97956bc7237bd"
Oct 29 23:03:14.295726 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.295657    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"
Oct 29 23:03:14.313383 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.311072    4559 scope.go:122] "RemoveContainer" containerID="6ca7c57a1b042aebfaae4b0fc22e934591411b859f3c7ae4b4e97956bc7237bd"
Oct 29 23:03:14.313383 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:14.313261    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca7c57a1b042aebfaae4b0fc22e934591411b859f3c7ae4b4e97956bc7237bd\": container with ID starting with 6ca7c57a1b042aebfaae4b0fc22e934591411b859f3c7ae4b4e97956bc7237bd not found: ID does not exist" containerID="6ca7c57a1b042aebfaae4b0fc22e934591411b859f3c7ae4b4e97956bc7237bd"
Oct 29 23:03:14.313383 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.313303    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca7c57a1b042aebfaae4b0fc22e934591411b859f3c7ae4b4e97956bc7237bd"} err="failed to get container status \"6ca7c57a1b042aebfaae4b0fc22e934591411b859f3c7ae4b4e97956bc7237bd\": rpc error: code = NotFound desc = could not find container \"6ca7c57a1b042aebfaae4b0fc22e934591411b859f3c7ae4b4e97956bc7237bd\": container with ID starting with 6ca7c57a1b042aebfaae4b0fc22e934591411b859f3c7ae4b4e97956bc7237bd not found: ID does not exist"
Oct 29 23:03:14.364084 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.361282    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"]
Oct 29 23:03:14.364084 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.361540    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt" podUID="2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" containerName="istio-proxy" containerID="cri-o://acec9394fbd0aa8edb2cd77af8478809f0c45ad81c609adc8e3e9d20c688c938" gracePeriod=30
Oct 29 23:03:14.411157 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.410267    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-openshift-service-ca\") pod \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\" (UID: \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\") "
Oct 29 23:03:14.411157 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.410369    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-kube-api-access-tg7xq\" (UniqueName: \"kubernetes.io/projected/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-kube-api-access-tg7xq\") pod \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\" (UID: \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\") "
Oct 29 23:03:14.411157 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.410443    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-metrics-certs\") pod \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\" (UID: \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\") "
Oct 29 23:03:14.411157 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.410544    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-maas-api-tls\") pod \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\" (UID: \"a1040d20-45e0-4df0-bc5a-2ca67e5e5108\") "
Oct 29 23:03:14.411877 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.411677    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-openshift-service-ca" pod "a1040d20-45e0-4df0-bc5a-2ca67e5e5108" (UID: "a1040d20-45e0-4df0-bc5a-2ca67e5e5108"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:14.417503 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.417473    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-kube-api-access-tg7xq" pod "a1040d20-45e0-4df0-bc5a-2ca67e5e5108" (UID: "a1040d20-45e0-4df0-bc5a-2ca67e5e5108"). InnerVolumeSpecName "kube-api-access-tg7xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:14.425054 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.425018    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-maas-api-tls" pod "a1040d20-45e0-4df0-bc5a-2ca67e5e5108" (UID: "a1040d20-45e0-4df0-bc5a-2ca67e5e5108"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:14.430775 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.430733    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-metrics-certs" pod "a1040d20-45e0-4df0-bc5a-2ca67e5e5108" (UID: "a1040d20-45e0-4df0-bc5a-2ca67e5e5108"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:14.472612 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.472586    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"
Oct 29 23:03:14.495997 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.495959    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr" podUID="341bcfef-7b8c-4732-a60c-2849614df3b7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:14.513329 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.513294    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.513329 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.513329    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg7xq\" (UniqueName: \"kubernetes.io/projected/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-kube-api-access-tg7xq\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.513329 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.513347    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.513329 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.513363    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1040d20-45e0-4df0-bc5a-2ca67e5e5108-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.624683 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.623298    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7776b504-8e2a-4138-b03a-60b47a61bd53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7776b504-8e2a-4138-b03a-60b47a61bd53-plugins-config-volume\") pod \"7776b504-8e2a-4138-b03a-60b47a61bd53\" (UID: \"7776b504-8e2a-4138-b03a-60b47a61bd53\") "
Oct 29 23:03:14.624683 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.623406    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7776b504-8e2a-4138-b03a-60b47a61bd53-kube-api-access-n2w65\" (UniqueName: \"kubernetes.io/projected/7776b504-8e2a-4138-b03a-60b47a61bd53-kube-api-access-n2w65\") pod \"7776b504-8e2a-4138-b03a-60b47a61bd53\" (UID: \"7776b504-8e2a-4138-b03a-60b47a61bd53\") "
Oct 29 23:03:14.641131 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.641080    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7776b504-8e2a-4138-b03a-60b47a61bd53-plugins-config-volume" pod "7776b504-8e2a-4138-b03a-60b47a61bd53" (UID: "7776b504-8e2a-4138-b03a-60b47a61bd53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:14.668834 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.661006    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7776b504-8e2a-4138-b03a-60b47a61bd53-kube-api-access-n2w65" pod "7776b504-8e2a-4138-b03a-60b47a61bd53" (UID: "7776b504-8e2a-4138-b03a-60b47a61bd53"). InnerVolumeSpecName "kube-api-access-n2w65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:14.723487 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.723436    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"]
Oct 29 23:03:14.725643 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.725619    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7776b504-8e2a-4138-b03a-60b47a61bd53-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.725710 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.725645    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2w65\" (UniqueName: \"kubernetes.io/projected/7776b504-8e2a-4138-b03a-60b47a61bd53-kube-api-access-n2w65\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.726681 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:14.726656    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-9aafe6-79d65bdf4b-4t4n2"]
Oct 29 23:03:15.049079 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.049038    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"]
Oct 29 23:03:15.049643 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.049627    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a1040d20-45e0-4df0-bc5a-2ca67e5e5108" containerName="maas-api"
Oct 29 23:03:15.049703 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.049645    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1040d20-45e0-4df0-bc5a-2ca67e5e5108" containerName="maas-api"
Oct 29 23:03:15.049703 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.049665    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e09d2003-0997-418a-8fa3-f64c92d28572" containerName="istio-proxy"
Oct 29 23:03:15.049703 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.049670    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09d2003-0997-418a-8fa3-f64c92d28572" containerName="istio-proxy"
Oct 29 23:03:15.049703 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.049701    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7776b504-8e2a-4138-b03a-60b47a61bd53" containerName="payload-processing"
Oct 29 23:03:15.049902 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.049706    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7776b504-8e2a-4138-b03a-60b47a61bd53" containerName="payload-processing"
Oct 29 23:03:15.049902 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.049719    4559 cpu_manager.go:414] "containerMap: removing container" podUID="29b7beb6-4444-4752-9f17-c5b76b8eb2f9" containerName="istio-proxy"
Oct 29 23:03:15.049902 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.049724    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b7beb6-4444-4752-9f17-c5b76b8eb2f9" containerName="istio-proxy"
Oct 29 23:03:15.049902 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.049799    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7776b504-8e2a-4138-b03a-60b47a61bd53" containerName="payload-processing"
Oct 29 23:03:15.049902 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.049827    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b7beb6-4444-4752-9f17-c5b76b8eb2f9" containerName="istio-proxy"
Oct 29 23:03:15.049902 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.049834    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09d2003-0997-418a-8fa3-f64c92d28572" containerName="istio-proxy"
Oct 29 23:03:15.049902 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.049843    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1040d20-45e0-4df0-bc5a-2ca67e5e5108" containerName="maas-api"
Oct 29 23:03:15.049902 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.049852    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09d2003-0997-418a-8fa3-f64c92d28572" containerName="istio-proxy"
Oct 29 23:03:15.053524 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.053506    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"
Oct 29 23:03:15.073522 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.073483    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"]
Oct 29 23:03:15.094028 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.093997    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:03:15.094231 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.094038    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:03:15.228453 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.228420    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1040d20-45e0-4df0-bc5a-2ca67e5e5108" path="/var/lib/kubelet/pods/a1040d20-45e0-4df0-bc5a-2ca67e5e5108/volumes"
Oct 29 23:03:15.231973 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.231951    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16a423d7-c2f3-4fc3-809b-debb115ff543-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk\" (UID: \"16a423d7-c2f3-4fc3-809b-debb115ff543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"
Oct 29 23:03:15.232133 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.232109    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqd8z\" (UniqueName: \"kubernetes.io/projected/16a423d7-c2f3-4fc3-809b-debb115ff543-kube-api-access-rqd8z\") pod \"maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk\" (UID: \"16a423d7-c2f3-4fc3-809b-debb115ff543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"
Oct 29 23:03:15.232229 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.232213    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16a423d7-c2f3-4fc3-809b-debb115ff543-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk\" (UID: \"16a423d7-c2f3-4fc3-809b-debb115ff543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"
Oct 29 23:03:15.305039 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.304938    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh" event={"ID":"7776b504-8e2a-4138-b03a-60b47a61bd53","Type":"ContainerDied","Data":"7e91605dc55e843a62eef0b018cdcf5e0b99d5d67ab76dd59cc486ebd5bd4605"}
Oct 29 23:03:15.305210 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.304953    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"
Oct 29 23:03:15.305210 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.305049    4559 scope.go:122] "RemoveContainer" containerID="3e30dfe662325dceb7fedc1bf7e7366c53ab76d86d12dbd752dc966a5245e554"
Oct 29 23:03:15.333427 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.333380    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16a423d7-c2f3-4fc3-809b-debb115ff543-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk\" (UID: \"16a423d7-c2f3-4fc3-809b-debb115ff543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"
Oct 29 23:03:15.333625 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.333556    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqd8z\" (UniqueName: \"kubernetes.io/projected/16a423d7-c2f3-4fc3-809b-debb115ff543-kube-api-access-rqd8z\") pod \"maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk\" (UID: \"16a423d7-c2f3-4fc3-809b-debb115ff543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"
Oct 29 23:03:15.333765 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.333745    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16a423d7-c2f3-4fc3-809b-debb115ff543-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk\" (UID: \"16a423d7-c2f3-4fc3-809b-debb115ff543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"
Oct 29 23:03:15.334440 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.334418    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16a423d7-c2f3-4fc3-809b-debb115ff543-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk\" (UID: \"16a423d7-c2f3-4fc3-809b-debb115ff543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"
Oct 29 23:03:15.336102 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.336077    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16a423d7-c2f3-4fc3-809b-debb115ff543-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk\" (UID: \"16a423d7-c2f3-4fc3-809b-debb115ff543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"
Oct 29 23:03:15.353756 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.353719    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqd8z\" (UniqueName: \"kubernetes.io/projected/16a423d7-c2f3-4fc3-809b-debb115ff543-kube-api-access-rqd8z\") pod \"maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk\" (UID: \"16a423d7-c2f3-4fc3-809b-debb115ff543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"
Oct 29 23:03:15.357108 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.357074    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"]
Oct 29 23:03:15.366726 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.366694    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-02720a-78cdf6c4cd-w4jmh"]
Oct 29 23:03:15.368454 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.368433    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"
Oct 29 23:03:15.676677 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:03:15.676632    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a423d7_c2f3_4fc3_809b_debb115ff543.slice/crio-f90288c0bb06b49eec244293a017a52132b359cb12b5e1a4f76081855c9f866b WatchSource:0}: Error finding container f90288c0bb06b49eec244293a017a52132b359cb12b5e1a4f76081855c9f866b: Status 404 returned error can't find the container with id f90288c0bb06b49eec244293a017a52132b359cb12b5e1a4f76081855c9f866b
Oct 29 23:03:15.677420 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.677391    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"]
Oct 29 23:03:15.902041 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:15.901999    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"]
Oct 29 23:03:16.315209 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:16.315111    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk" event={"ID":"16a423d7-c2f3-4fc3-809b-debb115ff543","Type":"ContainerStarted","Data":"77f6b76af1bfc6362c702ed13ead796df108398b48b4a9417aa9ed14377d1787"}
Oct 29 23:03:16.315209 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:16.315175    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk" event={"ID":"16a423d7-c2f3-4fc3-809b-debb115ff543","Type":"ContainerStarted","Data":"f90288c0bb06b49eec244293a017a52132b359cb12b5e1a4f76081855c9f866b"}
Oct 29 23:03:16.357229 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:16.357169    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk" podStartSLOduration=1.3571533 podStartE2EDuration="1.3571533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:16.354058865 +0000 UTC m=+2935.265502419" watchObservedRunningTime="2026-10-29 23:03:16.3571533 +0000 UTC m=+2935.268596853"
Oct 29 23:03:17.227931 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:17.227896    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7776b504-8e2a-4138-b03a-60b47a61bd53" path="/var/lib/kubelet/pods/7776b504-8e2a-4138-b03a-60b47a61bd53/volumes"
Oct 29 23:03:17.323949 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:17.323910    4559 generic.go:356] "Generic (PLEG): container finished" podID="16a423d7-c2f3-4fc3-809b-debb115ff543" containerID="77f6b76af1bfc6362c702ed13ead796df108398b48b4a9417aa9ed14377d1787" exitCode=0
Oct 29 23:03:17.324161 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:17.323974    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk" event={"ID":"16a423d7-c2f3-4fc3-809b-debb115ff543","Type":"ContainerDied","Data":"77f6b76af1bfc6362c702ed13ead796df108398b48b4a9417aa9ed14377d1787"}
Oct 29 23:03:17.991608 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:17.991575    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q_303751f6-3328-401c-9f2c-4970a54ae09c/payload-processing/0.log"
Oct 29 23:03:18.577484 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:18.577456    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"
Oct 29 23:03:18.588029 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:18.587945    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16a423d7-c2f3-4fc3-809b-debb115ff543-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16a423d7-c2f3-4fc3-809b-debb115ff543-maas-api-openshift-ca\") pod \"16a423d7-c2f3-4fc3-809b-debb115ff543\" (UID: \"16a423d7-c2f3-4fc3-809b-debb115ff543\") "
Oct 29 23:03:18.588029 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:18.587988    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16a423d7-c2f3-4fc3-809b-debb115ff543-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16a423d7-c2f3-4fc3-809b-debb115ff543-maas-api-xks-ca\") pod \"16a423d7-c2f3-4fc3-809b-debb115ff543\" (UID: \"16a423d7-c2f3-4fc3-809b-debb115ff543\") "
Oct 29 23:03:18.588268 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:18.588146    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a423d7-c2f3-4fc3-809b-debb115ff543-kube-api-access-rqd8z\" (UniqueName: \"kubernetes.io/projected/16a423d7-c2f3-4fc3-809b-debb115ff543-kube-api-access-rqd8z\") pod \"16a423d7-c2f3-4fc3-809b-debb115ff543\" (UID: \"16a423d7-c2f3-4fc3-809b-debb115ff543\") "
Oct 29 23:03:18.588374 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:18.588345    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16a423d7-c2f3-4fc3-809b-debb115ff543-maas-api-openshift-ca" pod "16a423d7-c2f3-4fc3-809b-debb115ff543" (UID: "16a423d7-c2f3-4fc3-809b-debb115ff543"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:18.588664 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:18.588643    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16a423d7-c2f3-4fc3-809b-debb115ff543-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:18.592706 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:18.592107    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a423d7-c2f3-4fc3-809b-debb115ff543-kube-api-access-rqd8z" pod "16a423d7-c2f3-4fc3-809b-debb115ff543" (UID: "16a423d7-c2f3-4fc3-809b-debb115ff543"). InnerVolumeSpecName "kube-api-access-rqd8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:18.592706 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:18.592202    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16a423d7-c2f3-4fc3-809b-debb115ff543-maas-api-xks-ca" pod "16a423d7-c2f3-4fc3-809b-debb115ff543" (UID: "16a423d7-c2f3-4fc3-809b-debb115ff543"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:18.689656 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:18.689622    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16a423d7-c2f3-4fc3-809b-debb115ff543-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:18.689854 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:18.689675    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqd8z\" (UniqueName: \"kubernetes.io/projected/16a423d7-c2f3-4fc3-809b-debb115ff543-kube-api-access-rqd8z\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:18.732853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:18.732802    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"
Oct 29 23:03:18.892288 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:18.892260    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d190385b-94ea-4b39-9318-3a33e14e8286-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d190385b-94ea-4b39-9318-3a33e14e8286-plugins-config-volume\") pod \"d190385b-94ea-4b39-9318-3a33e14e8286\" (UID: \"d190385b-94ea-4b39-9318-3a33e14e8286\") "
Oct 29 23:03:18.892464 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:18.892349    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d190385b-94ea-4b39-9318-3a33e14e8286-kube-api-access-jlx4d\" (UniqueName: \"kubernetes.io/projected/d190385b-94ea-4b39-9318-3a33e14e8286-kube-api-access-jlx4d\") pod \"d190385b-94ea-4b39-9318-3a33e14e8286\" (UID: \"d190385b-94ea-4b39-9318-3a33e14e8286\") "
Oct 29 23:03:18.892721 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:18.892692    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d190385b-94ea-4b39-9318-3a33e14e8286-plugins-config-volume" pod "d190385b-94ea-4b39-9318-3a33e14e8286" (UID: "d190385b-94ea-4b39-9318-3a33e14e8286"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:18.894601 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:18.894581    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d190385b-94ea-4b39-9318-3a33e14e8286-kube-api-access-jlx4d" pod "d190385b-94ea-4b39-9318-3a33e14e8286" (UID: "d190385b-94ea-4b39-9318-3a33e14e8286"). InnerVolumeSpecName "kube-api-access-jlx4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:18.993950 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:18.993909    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlx4d\" (UniqueName: \"kubernetes.io/projected/d190385b-94ea-4b39-9318-3a33e14e8286-kube-api-access-jlx4d\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:18.993950 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:18.993942    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d190385b-94ea-4b39-9318-3a33e14e8286-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.342253 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.342223    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"
Oct 29 23:03:19.342504 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.342254    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk" event={"ID":"16a423d7-c2f3-4fc3-809b-debb115ff543","Type":"ContainerDied","Data":"f90288c0bb06b49eec244293a017a52132b359cb12b5e1a4f76081855c9f866b"}
Oct 29 23:03:19.342504 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.342312    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f90288c0bb06b49eec244293a017a52132b359cb12b5e1a4f76081855c9f866b"
Oct 29 23:03:19.343939 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.343914    4559 generic.go:356] "Generic (PLEG): container finished" podID="d190385b-94ea-4b39-9318-3a33e14e8286" containerID="6f11f03f3179b369d195ccfa03394f3f21a5e21315a56272e5d455f1abf2a78d" exitCode=0
Oct 29 23:03:19.344082 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.343958    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"
Oct 29 23:03:19.344082 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.344005    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq" event={"ID":"d190385b-94ea-4b39-9318-3a33e14e8286","Type":"ContainerDied","Data":"6f11f03f3179b369d195ccfa03394f3f21a5e21315a56272e5d455f1abf2a78d"}
Oct 29 23:03:19.344082 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.344060    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq" event={"ID":"d190385b-94ea-4b39-9318-3a33e14e8286","Type":"ContainerDied","Data":"e02af42845c09ad219966e2200433582ddc9c185d78e9a288ae64e9cc8560ff0"}
Oct 29 23:03:19.344275 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.344087    4559 scope.go:122] "RemoveContainer" containerID="6f11f03f3179b369d195ccfa03394f3f21a5e21315a56272e5d455f1abf2a78d"
Oct 29 23:03:19.356490 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.356467    4559 scope.go:122] "RemoveContainer" containerID="6f11f03f3179b369d195ccfa03394f3f21a5e21315a56272e5d455f1abf2a78d"
Oct 29 23:03:19.356902 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:19.356851    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f11f03f3179b369d195ccfa03394f3f21a5e21315a56272e5d455f1abf2a78d\": container with ID starting with 6f11f03f3179b369d195ccfa03394f3f21a5e21315a56272e5d455f1abf2a78d not found: ID does not exist" containerID="6f11f03f3179b369d195ccfa03394f3f21a5e21315a56272e5d455f1abf2a78d"
Oct 29 23:03:19.357010 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.356892    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f11f03f3179b369d195ccfa03394f3f21a5e21315a56272e5d455f1abf2a78d"} err="failed to get container status \"6f11f03f3179b369d195ccfa03394f3f21a5e21315a56272e5d455f1abf2a78d\": rpc error: code = NotFound desc = could not find container \"6f11f03f3179b369d195ccfa03394f3f21a5e21315a56272e5d455f1abf2a78d\": container with ID starting with 6f11f03f3179b369d195ccfa03394f3f21a5e21315a56272e5d455f1abf2a78d not found: ID does not exist"
Oct 29 23:03:19.413030 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.412992    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"]
Oct 29 23:03:19.418675 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.418638    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3ee017-5f8b7645fb-477wq"]
Oct 29 23:03:19.652620 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.652586    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 23:03:19.701631 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.701599    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-workload-certs\") pod \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") "
Oct 29 23:03:19.701868 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.701672    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-envoy\") pod \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") "
Oct 29 23:03:19.701868 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.701694    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-credential-socket\") pod \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") "
Oct 29 23:03:19.701868 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.701738    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-kube-api-access-7cccl\" (UniqueName: \"kubernetes.io/projected/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-kube-api-access-7cccl\") pod \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") "
Oct 29 23:03:19.701868 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.701762    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-data\") pod \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") "
Oct 29 23:03:19.701868 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.701830    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-workload-socket\") pod \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") "
Oct 29 23:03:19.701868 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.701855    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istiod-ca-cert\") pod \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") "
Oct 29 23:03:19.702251 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.701976    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-token\" (UniqueName: \"kubernetes.io/projected/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-token\") pod \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") "
Oct 29 23:03:19.702251 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.702018    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-podinfo\") pod \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\" (UID: \"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d\") "
Oct 29 23:03:19.703031 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.702759    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-workload-socket" pod "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" (UID: "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:19.703031 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.702781    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-workload-certs" pod "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" (UID: "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:19.703240 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.703048    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-data" pod "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" (UID: "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:19.703240 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.703061    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-credential-socket" pod "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" (UID: "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:19.703491 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.703361    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istiod-ca-cert" pod "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" (UID: "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:19.706554 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.706522    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-podinfo" pod "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" (UID: "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:19.707071 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.707039    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-token" pod "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" (UID: "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:19.707506 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.707466    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-envoy" pod "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" (UID: "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:19.708030 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.708004    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-kube-api-access-7cccl" pod "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" (UID: "2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d"). InnerVolumeSpecName "kube-api-access-7cccl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:19.803382 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.803292    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.803382 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.803326    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.803382 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.803335    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.803382 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.803342    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.803382 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.803350    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.803382 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.803358    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cccl\" (UniqueName: \"kubernetes.io/projected/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-kube-api-access-7cccl\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.803382 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.803367    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.803382 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.803377    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.803382 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.803385    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.857471 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:19.857434    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"]
Oct 29 23:03:20.352910 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:20.352870    4559 generic.go:356] "Generic (PLEG): container finished" podID="2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" containerID="acec9394fbd0aa8edb2cd77af8478809f0c45ad81c609adc8e3e9d20c688c938" exitCode=0
Oct 29 23:03:20.353114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:20.352924    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"
Oct 29 23:03:20.353114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:20.352955    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt" event={"ID":"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d","Type":"ContainerDied","Data":"acec9394fbd0aa8edb2cd77af8478809f0c45ad81c609adc8e3e9d20c688c938"}
Oct 29 23:03:20.353114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:20.353004    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt" event={"ID":"2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d","Type":"ContainerDied","Data":"0ce837734b8249bcf7fa631241023d65820e2b652aa4d139986cd5fbfaa6a961"}
Oct 29 23:03:20.353114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:20.353028    4559 scope.go:122] "RemoveContainer" containerID="acec9394fbd0aa8edb2cd77af8478809f0c45ad81c609adc8e3e9d20c688c938"
Oct 29 23:03:20.365553 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:20.365527    4559 scope.go:122] "RemoveContainer" containerID="acec9394fbd0aa8edb2cd77af8478809f0c45ad81c609adc8e3e9d20c688c938"
Oct 29 23:03:20.365945 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:20.365920    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acec9394fbd0aa8edb2cd77af8478809f0c45ad81c609adc8e3e9d20c688c938\": container with ID starting with acec9394fbd0aa8edb2cd77af8478809f0c45ad81c609adc8e3e9d20c688c938 not found: ID does not exist" containerID="acec9394fbd0aa8edb2cd77af8478809f0c45ad81c609adc8e3e9d20c688c938"
Oct 29 23:03:20.366018 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:20.365956    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acec9394fbd0aa8edb2cd77af8478809f0c45ad81c609adc8e3e9d20c688c938"} err="failed to get container status \"acec9394fbd0aa8edb2cd77af8478809f0c45ad81c609adc8e3e9d20c688c938\": rpc error: code = NotFound desc = could not find container \"acec9394fbd0aa8edb2cd77af8478809f0c45ad81c609adc8e3e9d20c688c938\": container with ID starting with acec9394fbd0aa8edb2cd77af8478809f0c45ad81c609adc8e3e9d20c688c938 not found: ID does not exist"
Oct 29 23:03:20.407893 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:20.407854    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"]
Oct 29 23:03:20.423228 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:20.423188    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-9m5dt"]
Oct 29 23:03:20.902572 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:20.902507    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7" podUID="b67ef029-68f6-4d3e-acf1-6637d11c9ae2" containerName="payload-pre-processing" containerID="cri-o://b083dc8de6a296a7eacd128265a4e015bcac2836569d28d0a70be40a3ba29603" gracePeriod=25
Oct 29 23:03:21.229005 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:21.228929    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" path="/var/lib/kubelet/pods/2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d/volumes"
Oct 29 23:03:21.229536 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:21.229520    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d190385b-94ea-4b39-9318-3a33e14e8286" path="/var/lib/kubelet/pods/d190385b-94ea-4b39-9318-3a33e14e8286/volumes"
Oct 29 23:03:21.771540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:21.771483    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"]
Oct 29 23:03:21.771829 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:21.771780    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc" podUID="e574d0f5-9916-4d71-9d2c-8743336447b3" containerName="istio-proxy" containerID="cri-o://a8a480dbd7be7173ae310afba23046d23b49bb9340506603150ffc180ba42c48" gracePeriod=30
Oct 29 23:03:22.311782 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.311744    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"]
Oct 29 23:03:22.312348 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.312331    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d190385b-94ea-4b39-9318-3a33e14e8286" containerName="payload-processing"
Oct 29 23:03:22.312348 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.312348    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d190385b-94ea-4b39-9318-3a33e14e8286" containerName="payload-processing"
Oct 29 23:03:22.312445 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.312369    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" containerName="istio-proxy"
Oct 29 23:03:22.312445 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.312374    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" containerName="istio-proxy"
Oct 29 23:03:22.312445 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.312387    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e09d2003-0997-418a-8fa3-f64c92d28572" containerName="istio-proxy"
Oct 29 23:03:22.312445 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.312393    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09d2003-0997-418a-8fa3-f64c92d28572" containerName="istio-proxy"
Oct 29 23:03:22.312445 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.312413    4559 cpu_manager.go:414] "containerMap: removing container" podUID="16a423d7-c2f3-4fc3-809b-debb115ff543" containerName="revoke-keys"
Oct 29 23:03:22.312445 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.312418    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a423d7-c2f3-4fc3-809b-debb115ff543" containerName="revoke-keys"
Oct 29 23:03:22.312668 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.312505    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a423d7-c2f3-4fc3-809b-debb115ff543" containerName="revoke-keys"
Oct 29 23:03:22.312668 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.312517    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d190385b-94ea-4b39-9318-3a33e14e8286" containerName="payload-processing"
Oct 29 23:03:22.312668 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.312525    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a55532b-2fce-4bce-9b7b-9ef09bd7aa3d" containerName="istio-proxy"
Oct 29 23:03:22.316370 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.316348    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"
Oct 29 23:03:22.332569 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.332534    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmmnl\" (UniqueName: \"kubernetes.io/projected/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-kube-api-access-kmmnl\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q\" (UID: \"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"
Oct 29 23:03:22.332787 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.332734    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q\" (UID: \"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"
Oct 29 23:03:22.332893 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.332792    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q\" (UID: \"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"
Oct 29 23:03:22.337054 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.337018    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"]
Oct 29 23:03:22.351743 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.351711    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:03:22.351937 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.351833    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:03:22.438930 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.438895    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q\" (UID: \"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"
Oct 29 23:03:22.439124 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.438968    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q\" (UID: \"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"
Oct 29 23:03:22.439124 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.439018    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmmnl\" (UniqueName: \"kubernetes.io/projected/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-kube-api-access-kmmnl\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q\" (UID: \"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"
Oct 29 23:03:22.439623 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.439598    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q\" (UID: \"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"
Oct 29 23:03:22.441736 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.441708    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q\" (UID: \"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"
Oct 29 23:03:22.459000 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.458969    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmmnl\" (UniqueName: \"kubernetes.io/projected/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-kube-api-access-kmmnl\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q\" (UID: \"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"
Oct 29 23:03:22.482356 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.482321    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h" podUID="299a03af-cb40-46b1-b42d-8afa20c7cd00" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:22.630856 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.630754    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"
Oct 29 23:03:22.933995 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:22.933948    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"]
Oct 29 23:03:22.950103 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:03:22.949953    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a0cfb5_901f_4734_bd40_e6a9f0f53e0f.slice/crio-2ea26288645549f2d8ada56303ece9c8b3d34a46c4473d740cc1c5b7cfe6aff6 WatchSource:0}: Error finding container 2ea26288645549f2d8ada56303ece9c8b3d34a46c4473d740cc1c5b7cfe6aff6: Status 404 returned error can't find the container with id 2ea26288645549f2d8ada56303ece9c8b3d34a46c4473d740cc1c5b7cfe6aff6
Oct 29 23:03:23.383120 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:23.383078    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q" event={"ID":"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f","Type":"ContainerStarted","Data":"c8261afece87aeb734717c2eea8ab1651392136ce678d6cba4337e3a590466bf"}
Oct 29 23:03:23.383558 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:23.383140    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q" event={"ID":"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f","Type":"ContainerStarted","Data":"2ea26288645549f2d8ada56303ece9c8b3d34a46c4473d740cc1c5b7cfe6aff6"}
Oct 29 23:03:23.422829 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:23.422762    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q" podStartSLOduration=1.4227468110000001 podStartE2EDuration="1.422746811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:23.421226818 +0000 UTC m=+2942.332670372" watchObservedRunningTime="2026-10-29 23:03:23.422746811 +0000 UTC m=+2942.334190371"
Oct 29 23:03:23.427138 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:23.427093    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:03:23.427362 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:23.427158    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc" podUID="e574d0f5-9916-4d71-9d2c-8743336447b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:03:24.391993 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:24.391950    4559 generic.go:356] "Generic (PLEG): container finished" podID="04a0cfb5-901f-4734-bd40-e6a9f0f53e0f" containerID="c8261afece87aeb734717c2eea8ab1651392136ce678d6cba4337e3a590466bf" exitCode=0
Oct 29 23:03:24.392417 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:24.392096    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q" event={"ID":"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f","Type":"ContainerDied","Data":"c8261afece87aeb734717c2eea8ab1651392136ce678d6cba4337e3a590466bf"}
Oct 29 23:03:24.496686 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:24.496649    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr" podUID="341bcfef-7b8c-4732-a60c-2849614df3b7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:25.258603 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.258551    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" containerID="cri-o://4e618b766aed8eea4c86ec3fbb1c2d27922409f926fc50a154a074c3b62014da" gracePeriod=45
Oct 29 23:03:25.401831 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.401773    4559 generic.go:356] "Generic (PLEG): container finished" podID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerID="4e618b766aed8eea4c86ec3fbb1c2d27922409f926fc50a154a074c3b62014da" exitCode=0
Oct 29 23:03:25.402355 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.401855    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" event={"ID":"154272dc-9da5-4f5d-ab1f-54282fe184d3","Type":"ContainerDied","Data":"4e618b766aed8eea4c86ec3fbb1c2d27922409f926fc50a154a074c3b62014da"}
Oct 29 23:03:25.552600 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.552542    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"]
Oct 29 23:03:25.571979 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.571946    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 23:03:25.628292 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.628265    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"
Oct 29 23:03:25.680618 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.680498    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154272dc-9da5-4f5d-ab1f-54282fe184d3-kube-api-access-749dt\" (UniqueName: \"kubernetes.io/projected/154272dc-9da5-4f5d-ab1f-54282fe184d3-kube-api-access-749dt\") pod \"154272dc-9da5-4f5d-ab1f-54282fe184d3\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") "
Oct 29 23:03:25.680618 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.680550    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-dshm\") pod \"154272dc-9da5-4f5d-ab1f-54282fe184d3\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") "
Oct 29 23:03:25.680618 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.680600    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/154272dc-9da5-4f5d-ab1f-54282fe184d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/154272dc-9da5-4f5d-ab1f-54282fe184d3-tls-certs\") pod \"154272dc-9da5-4f5d-ab1f-54282fe184d3\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") "
Oct 29 23:03:25.681087 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.680760    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-tmp-dir\") pod \"154272dc-9da5-4f5d-ab1f-54282fe184d3\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") "
Oct 29 23:03:25.681087 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.680860    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-home\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-home\") pod \"154272dc-9da5-4f5d-ab1f-54282fe184d3\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") "
Oct 29 23:03:25.681087 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.680899    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-model-cache\") pod \"154272dc-9da5-4f5d-ab1f-54282fe184d3\" (UID: \"154272dc-9da5-4f5d-ab1f-54282fe184d3\") "
Oct 29 23:03:25.686169 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.681642    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-model-cache" pod "154272dc-9da5-4f5d-ab1f-54282fe184d3" (UID: "154272dc-9da5-4f5d-ab1f-54282fe184d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:25.686169 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.684128    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-home" pod "154272dc-9da5-4f5d-ab1f-54282fe184d3" (UID: "154272dc-9da5-4f5d-ab1f-54282fe184d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:25.686169 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.684278    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-tmp-dir" pod "154272dc-9da5-4f5d-ab1f-54282fe184d3" (UID: "154272dc-9da5-4f5d-ab1f-54282fe184d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:25.686169 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.685780    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/154272dc-9da5-4f5d-ab1f-54282fe184d3-tls-certs" pod "154272dc-9da5-4f5d-ab1f-54282fe184d3" (UID: "154272dc-9da5-4f5d-ab1f-54282fe184d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:25.686169 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.685980    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154272dc-9da5-4f5d-ab1f-54282fe184d3-kube-api-access-749dt" pod "154272dc-9da5-4f5d-ab1f-54282fe184d3" (UID: "154272dc-9da5-4f5d-ab1f-54282fe184d3"). InnerVolumeSpecName "kube-api-access-749dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:25.687461 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.687420    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-dshm" pod "154272dc-9da5-4f5d-ab1f-54282fe184d3" (UID: "154272dc-9da5-4f5d-ab1f-54282fe184d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:25.799455 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.792960    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-maas-api-openshift-ca\") pod \"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f\" (UID: \"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f\") "
Oct 29 23:03:25.799455 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.793071    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-kube-api-access-kmmnl\" (UniqueName: \"kubernetes.io/projected/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-kube-api-access-kmmnl\") pod \"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f\" (UID: \"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f\") "
Oct 29 23:03:25.799455 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.793232    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-maas-api-xks-ca\") pod \"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f\" (UID: \"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f\") "
Oct 29 23:03:25.799455 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.793869    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.799455 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.798286    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.799455 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.798315    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-749dt\" (UniqueName: \"kubernetes.io/projected/154272dc-9da5-4f5d-ab1f-54282fe184d3-kube-api-access-749dt\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.799455 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.798334    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.799455 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.798355    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154272dc-9da5-4f5d-ab1f-54282fe184d3-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.799455 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.798373    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154272dc-9da5-4f5d-ab1f-54282fe184d3-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.799455 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.794772    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-maas-api-openshift-ca" pod "04a0cfb5-901f-4734-bd40-e6a9f0f53e0f" (UID: "04a0cfb5-901f-4734-bd40-e6a9f0f53e0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:25.799455 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.799311    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-maas-api-xks-ca" pod "04a0cfb5-901f-4734-bd40-e6a9f0f53e0f" (UID: "04a0cfb5-901f-4734-bd40-e6a9f0f53e0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:25.804882 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.804845    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-kube-api-access-kmmnl" pod "04a0cfb5-901f-4734-bd40-e6a9f0f53e0f" (UID: "04a0cfb5-901f-4734-bd40-e6a9f0f53e0f"). InnerVolumeSpecName "kube-api-access-kmmnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:25.899568 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.899534    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.899568 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.899565    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmmnl\" (UniqueName: \"kubernetes.io/projected/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-kube-api-access-kmmnl\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:25.899568 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:25.899575    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:26.409904 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:26.409868    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"
Oct 29 23:03:26.410380 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:26.409906    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" event={"ID":"154272dc-9da5-4f5d-ab1f-54282fe184d3","Type":"ContainerDied","Data":"6ee10e73f7ab12f7f8a2614f6be0a6e6fdfce3b6883a8021626a2a4ec0cc7309"}
Oct 29 23:03:26.410380 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:26.409956    4559 scope.go:122] "RemoveContainer" containerID="4e618b766aed8eea4c86ec3fbb1c2d27922409f926fc50a154a074c3b62014da"
Oct 29 23:03:26.411601 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:26.411581    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"
Oct 29 23:03:26.411707 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:26.411613    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q" event={"ID":"04a0cfb5-901f-4734-bd40-e6a9f0f53e0f","Type":"ContainerDied","Data":"2ea26288645549f2d8ada56303ece9c8b3d34a46c4473d740cc1c5b7cfe6aff6"}
Oct 29 23:03:26.411707 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:26.411667    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ea26288645549f2d8ada56303ece9c8b3d34a46c4473d740cc1c5b7cfe6aff6"
Oct 29 23:03:26.479548 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:26.479498    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"]
Oct 29 23:03:26.484880 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:26.484848    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn"]
Oct 29 23:03:26.566313 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:26.553912    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e046a/simulator-w0-2e046a-kserve-7df67475d7-x79mn" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:03:27.009591 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.009564    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"
Oct 29 23:03:27.011726 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.011688    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"
Oct 29 23:03:27.077188 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.077148    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 23:03:27.113355 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.113262    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341bcfef-7b8c-4732-a60c-2849614df3b7-kube-api-access-gmfvk\" (UniqueName: \"kubernetes.io/projected/341bcfef-7b8c-4732-a60c-2849614df3b7-kube-api-access-gmfvk\") pod \"341bcfef-7b8c-4732-a60c-2849614df3b7\" (UID: \"341bcfef-7b8c-4732-a60c-2849614df3b7\") "
Oct 29 23:03:27.113355 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.113338    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/299a03af-cb40-46b1-b42d-8afa20c7cd00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299a03af-cb40-46b1-b42d-8afa20c7cd00-plugins-config-volume\") pod \"299a03af-cb40-46b1-b42d-8afa20c7cd00\" (UID: \"299a03af-cb40-46b1-b42d-8afa20c7cd00\") "
Oct 29 23:03:27.113596 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.113377    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-credential-socket\") pod \"e574d0f5-9916-4d71-9d2c-8743336447b3\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") "
Oct 29 23:03:27.113596 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.113446    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-data\") pod \"e574d0f5-9916-4d71-9d2c-8743336447b3\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") "
Oct 29 23:03:27.113596 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.113475    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-workload-socket\") pod \"e574d0f5-9916-4d71-9d2c-8743336447b3\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") "
Oct 29 23:03:27.113596 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.113509    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e574d0f5-9916-4d71-9d2c-8743336447b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e574d0f5-9916-4d71-9d2c-8743336447b3-istiod-ca-cert\") pod \"e574d0f5-9916-4d71-9d2c-8743336447b3\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") "
Oct 29 23:03:27.113596 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.113537    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-workload-certs\") pod \"e574d0f5-9916-4d71-9d2c-8743336447b3\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") "
Oct 29 23:03:27.113918 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.113595    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e574d0f5-9916-4d71-9d2c-8743336447b3-kube-api-access-tth5p\" (UniqueName: \"kubernetes.io/projected/e574d0f5-9916-4d71-9d2c-8743336447b3-kube-api-access-tth5p\") pod \"e574d0f5-9916-4d71-9d2c-8743336447b3\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") "
Oct 29 23:03:27.113918 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.113649    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-podinfo\") pod \"e574d0f5-9916-4d71-9d2c-8743336447b3\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") "
Oct 29 23:03:27.113918 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.113699    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-envoy\") pod \"e574d0f5-9916-4d71-9d2c-8743336447b3\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") "
Oct 29 23:03:27.113918 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.113728    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/341bcfef-7b8c-4732-a60c-2849614df3b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341bcfef-7b8c-4732-a60c-2849614df3b7-plugins-config-volume\") pod \"341bcfef-7b8c-4732-a60c-2849614df3b7\" (UID: \"341bcfef-7b8c-4732-a60c-2849614df3b7\") "
Oct 29 23:03:27.113918 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.113775    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299a03af-cb40-46b1-b42d-8afa20c7cd00-kube-api-access-8lkch\" (UniqueName: \"kubernetes.io/projected/299a03af-cb40-46b1-b42d-8afa20c7cd00-kube-api-access-8lkch\") pod \"299a03af-cb40-46b1-b42d-8afa20c7cd00\" (UID: \"299a03af-cb40-46b1-b42d-8afa20c7cd00\") "
Oct 29 23:03:27.113918 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.113853    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-token\" (UniqueName: \"kubernetes.io/projected/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-token\") pod \"e574d0f5-9916-4d71-9d2c-8743336447b3\" (UID: \"e574d0f5-9916-4d71-9d2c-8743336447b3\") "
Oct 29 23:03:27.113918 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.113883    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-data" pod "e574d0f5-9916-4d71-9d2c-8743336447b3" (UID: "e574d0f5-9916-4d71-9d2c-8743336447b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:27.113918 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.113897    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-workload-socket" pod "e574d0f5-9916-4d71-9d2c-8743336447b3" (UID: "e574d0f5-9916-4d71-9d2c-8743336447b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:27.114359 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.114270    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341bcfef-7b8c-4732-a60c-2849614df3b7-plugins-config-volume" pod "341bcfef-7b8c-4732-a60c-2849614df3b7" (UID: "341bcfef-7b8c-4732-a60c-2849614df3b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:27.114415 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.114401    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.114471 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.114423    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.114471 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.114438    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341bcfef-7b8c-4732-a60c-2849614df3b7-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.114873 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.114845    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-workload-certs" pod "e574d0f5-9916-4d71-9d2c-8743336447b3" (UID: "e574d0f5-9916-4d71-9d2c-8743336447b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:27.114958 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.114912    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-credential-socket" pod "e574d0f5-9916-4d71-9d2c-8743336447b3" (UID: "e574d0f5-9916-4d71-9d2c-8743336447b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:27.115256 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.115232    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/299a03af-cb40-46b1-b42d-8afa20c7cd00-plugins-config-volume" pod "299a03af-cb40-46b1-b42d-8afa20c7cd00" (UID: "299a03af-cb40-46b1-b42d-8afa20c7cd00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:27.115349 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.115322    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e574d0f5-9916-4d71-9d2c-8743336447b3-istiod-ca-cert" pod "e574d0f5-9916-4d71-9d2c-8743336447b3" (UID: "e574d0f5-9916-4d71-9d2c-8743336447b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:27.117072 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.117040    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e574d0f5-9916-4d71-9d2c-8743336447b3-kube-api-access-tth5p" pod "e574d0f5-9916-4d71-9d2c-8743336447b3" (UID: "e574d0f5-9916-4d71-9d2c-8743336447b3"). InnerVolumeSpecName "kube-api-access-tth5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:27.117206 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.117137    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-token" pod "e574d0f5-9916-4d71-9d2c-8743336447b3" (UID: "e574d0f5-9916-4d71-9d2c-8743336447b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:27.117288 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.117266    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341bcfef-7b8c-4732-a60c-2849614df3b7-kube-api-access-gmfvk" pod "341bcfef-7b8c-4732-a60c-2849614df3b7" (UID: "341bcfef-7b8c-4732-a60c-2849614df3b7"). InnerVolumeSpecName "kube-api-access-gmfvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:27.117901 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.117875    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299a03af-cb40-46b1-b42d-8afa20c7cd00-kube-api-access-8lkch" pod "299a03af-cb40-46b1-b42d-8afa20c7cd00" (UID: "299a03af-cb40-46b1-b42d-8afa20c7cd00"). InnerVolumeSpecName "kube-api-access-8lkch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:27.118293 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.118270    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-podinfo" pod "e574d0f5-9916-4d71-9d2c-8743336447b3" (UID: "e574d0f5-9916-4d71-9d2c-8743336447b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:27.118872 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.118850    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-envoy" pod "e574d0f5-9916-4d71-9d2c-8743336447b3" (UID: "e574d0f5-9916-4d71-9d2c-8743336447b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:27.215982 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.215940    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.215982 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.215974    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmfvk\" (UniqueName: \"kubernetes.io/projected/341bcfef-7b8c-4732-a60c-2849614df3b7-kube-api-access-gmfvk\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.215982 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.215984    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299a03af-cb40-46b1-b42d-8afa20c7cd00-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.215982 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.215993    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.216238 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.216002    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e574d0f5-9916-4d71-9d2c-8743336447b3-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.216238 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.216013    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.216238 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.216022    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tth5p\" (UniqueName: \"kubernetes.io/projected/e574d0f5-9916-4d71-9d2c-8743336447b3-kube-api-access-tth5p\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.216238 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.216030    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.216238 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.216038    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e574d0f5-9916-4d71-9d2c-8743336447b3-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.216238 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.216045    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lkch\" (UniqueName: \"kubernetes.io/projected/299a03af-cb40-46b1-b42d-8afa20c7cd00-kube-api-access-8lkch\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.227942 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.227903    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" path="/var/lib/kubelet/pods/154272dc-9da5-4f5d-ab1f-54282fe184d3/volumes"
Oct 29 23:03:27.420700 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.420663    4559 generic.go:356] "Generic (PLEG): container finished" podID="e574d0f5-9916-4d71-9d2c-8743336447b3" containerID="a8a480dbd7be7173ae310afba23046d23b49bb9340506603150ffc180ba42c48" exitCode=0
Oct 29 23:03:27.421237 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.420714    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"
Oct 29 23:03:27.421237 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.420743    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc" event={"ID":"e574d0f5-9916-4d71-9d2c-8743336447b3","Type":"ContainerDied","Data":"a8a480dbd7be7173ae310afba23046d23b49bb9340506603150ffc180ba42c48"}
Oct 29 23:03:27.421237 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.420801    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc" event={"ID":"e574d0f5-9916-4d71-9d2c-8743336447b3","Type":"ContainerDied","Data":"87bc2e70b7315e862438f8ae835b3fcf077b1bba3565720d65ab475f980640a2"}
Oct 29 23:03:27.421237 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.420848    4559 scope.go:122] "RemoveContainer" containerID="a8a480dbd7be7173ae310afba23046d23b49bb9340506603150ffc180ba42c48"
Oct 29 23:03:27.422277 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.422251    4559 generic.go:356] "Generic (PLEG): container finished" podID="299a03af-cb40-46b1-b42d-8afa20c7cd00" containerID="f27e188cea94f371d194cfe0a0111d1ef6f7a5f44fa93061767d9fefd38867ae" exitCode=0
Oct 29 23:03:27.422406 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.422302    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"
Oct 29 23:03:27.422406 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.422374    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h" event={"ID":"299a03af-cb40-46b1-b42d-8afa20c7cd00","Type":"ContainerDied","Data":"f27e188cea94f371d194cfe0a0111d1ef6f7a5f44fa93061767d9fefd38867ae"}
Oct 29 23:03:27.422546 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.422436    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h" event={"ID":"299a03af-cb40-46b1-b42d-8afa20c7cd00","Type":"ContainerDied","Data":"de3e9e3df99f8c004e94ea154a04030f7358ec6a777c7cbfefe29a78c586b2d5"}
Oct 29 23:03:27.424229 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.424207    4559 generic.go:356] "Generic (PLEG): container finished" podID="341bcfef-7b8c-4732-a60c-2849614df3b7" containerID="86e6ce7b87b79adf712b19c0b2f6885ef9b5cf53e0028e85423f5ea71ac8ed15" exitCode=0
Oct 29 23:03:27.424340 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.424252    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"
Oct 29 23:03:27.424340 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.424295    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr" event={"ID":"341bcfef-7b8c-4732-a60c-2849614df3b7","Type":"ContainerDied","Data":"86e6ce7b87b79adf712b19c0b2f6885ef9b5cf53e0028e85423f5ea71ac8ed15"}
Oct 29 23:03:27.424466 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.424357    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr" event={"ID":"341bcfef-7b8c-4732-a60c-2849614df3b7","Type":"ContainerDied","Data":"d2a59da1b908c89f5e8828c46e3d1f3dac984e6707099cefb1f5edb5b3d9b33a"}
Oct 29 23:03:27.432212 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.432194    4559 scope.go:122] "RemoveContainer" containerID="a8a480dbd7be7173ae310afba23046d23b49bb9340506603150ffc180ba42c48"
Oct 29 23:03:27.432520 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:27.432500    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a480dbd7be7173ae310afba23046d23b49bb9340506603150ffc180ba42c48\": container with ID starting with a8a480dbd7be7173ae310afba23046d23b49bb9340506603150ffc180ba42c48 not found: ID does not exist" containerID="a8a480dbd7be7173ae310afba23046d23b49bb9340506603150ffc180ba42c48"
Oct 29 23:03:27.432561 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.432529    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a480dbd7be7173ae310afba23046d23b49bb9340506603150ffc180ba42c48"} err="failed to get container status \"a8a480dbd7be7173ae310afba23046d23b49bb9340506603150ffc180ba42c48\": rpc error: code = NotFound desc = could not find container \"a8a480dbd7be7173ae310afba23046d23b49bb9340506603150ffc180ba42c48\": container with ID starting with a8a480dbd7be7173ae310afba23046d23b49bb9340506603150ffc180ba42c48 not found: ID does not exist"
Oct 29 23:03:27.432561 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.432546    4559 scope.go:122] "RemoveContainer" containerID="f27e188cea94f371d194cfe0a0111d1ef6f7a5f44fa93061767d9fefd38867ae"
Oct 29 23:03:27.443940 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.443919    4559 scope.go:122] "RemoveContainer" containerID="f27e188cea94f371d194cfe0a0111d1ef6f7a5f44fa93061767d9fefd38867ae"
Oct 29 23:03:27.444281 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:27.444258    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27e188cea94f371d194cfe0a0111d1ef6f7a5f44fa93061767d9fefd38867ae\": container with ID starting with f27e188cea94f371d194cfe0a0111d1ef6f7a5f44fa93061767d9fefd38867ae not found: ID does not exist" containerID="f27e188cea94f371d194cfe0a0111d1ef6f7a5f44fa93061767d9fefd38867ae"
Oct 29 23:03:27.444345 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.444290    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27e188cea94f371d194cfe0a0111d1ef6f7a5f44fa93061767d9fefd38867ae"} err="failed to get container status \"f27e188cea94f371d194cfe0a0111d1ef6f7a5f44fa93061767d9fefd38867ae\": rpc error: code = NotFound desc = could not find container \"f27e188cea94f371d194cfe0a0111d1ef6f7a5f44fa93061767d9fefd38867ae\": container with ID starting with f27e188cea94f371d194cfe0a0111d1ef6f7a5f44fa93061767d9fefd38867ae not found: ID does not exist"
Oct 29 23:03:27.444345 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.444309    4559 scope.go:122] "RemoveContainer" containerID="86e6ce7b87b79adf712b19c0b2f6885ef9b5cf53e0028e85423f5ea71ac8ed15"
Oct 29 23:03:27.455659 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.455634    4559 scope.go:122] "RemoveContainer" containerID="86e6ce7b87b79adf712b19c0b2f6885ef9b5cf53e0028e85423f5ea71ac8ed15"
Oct 29 23:03:27.456021 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:27.456003    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e6ce7b87b79adf712b19c0b2f6885ef9b5cf53e0028e85423f5ea71ac8ed15\": container with ID starting with 86e6ce7b87b79adf712b19c0b2f6885ef9b5cf53e0028e85423f5ea71ac8ed15 not found: ID does not exist" containerID="86e6ce7b87b79adf712b19c0b2f6885ef9b5cf53e0028e85423f5ea71ac8ed15"
Oct 29 23:03:27.456073 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.456032    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e6ce7b87b79adf712b19c0b2f6885ef9b5cf53e0028e85423f5ea71ac8ed15"} err="failed to get container status \"86e6ce7b87b79adf712b19c0b2f6885ef9b5cf53e0028e85423f5ea71ac8ed15\": rpc error: code = NotFound desc = could not find container \"86e6ce7b87b79adf712b19c0b2f6885ef9b5cf53e0028e85423f5ea71ac8ed15\": container with ID starting with 86e6ce7b87b79adf712b19c0b2f6885ef9b5cf53e0028e85423f5ea71ac8ed15 not found: ID does not exist"
Oct 29 23:03:27.473692 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.473634    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"]
Oct 29 23:03:27.476546 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.476513    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9aafe6-788454f66b-t8w2h"]
Oct 29 23:03:27.514093 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.514052    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"]
Oct 29 23:03:27.516916 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.516888    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-9aafe6-6f8f97cff9-4cmpr"]
Oct 29 23:03:27.553046 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.553004    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"]
Oct 29 23:03:27.563840 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.559598    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-r9lfc"]
Oct 29 23:03:27.848273 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.848177    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"]
Oct 29 23:03:27.848754 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.848738    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e574d0f5-9916-4d71-9d2c-8743336447b3" containerName="istio-proxy"
Oct 29 23:03:27.848849 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.848754    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e574d0f5-9916-4d71-9d2c-8743336447b3" containerName="istio-proxy"
Oct 29 23:03:27.848849 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.848772    4559 cpu_manager.go:414] "containerMap: removing container" podUID="299a03af-cb40-46b1-b42d-8afa20c7cd00" containerName="payload-processing"
Oct 29 23:03:27.848849 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.848777    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299a03af-cb40-46b1-b42d-8afa20c7cd00" containerName="payload-processing"
Oct 29 23:03:27.848849 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.848794    4559 cpu_manager.go:414] "containerMap: removing container" podUID="341bcfef-7b8c-4732-a60c-2849614df3b7" containerName="payload-pre-processing"
Oct 29 23:03:27.848849 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.848800    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="341bcfef-7b8c-4732-a60c-2849614df3b7" containerName="payload-pre-processing"
Oct 29 23:03:27.848849 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.848832    4559 cpu_manager.go:414] "containerMap: removing container" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main"
Oct 29 23:03:27.848849 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.848839    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main"
Oct 29 23:03:27.849241 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.848858    4559 cpu_manager.go:414] "containerMap: removing container" podUID="04a0cfb5-901f-4734-bd40-e6a9f0f53e0f" containerName="revoke-keys"
Oct 29 23:03:27.849241 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.848863    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a0cfb5-901f-4734-bd40-e6a9f0f53e0f" containerName="revoke-keys"
Oct 29 23:03:27.849241 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.848942    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="299a03af-cb40-46b1-b42d-8afa20c7cd00" containerName="payload-processing"
Oct 29 23:03:27.849241 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.848951    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="341bcfef-7b8c-4732-a60c-2849614df3b7" containerName="payload-pre-processing"
Oct 29 23:03:27.849241 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.848959    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="154272dc-9da5-4f5d-ab1f-54282fe184d3" containerName="main"
Oct 29 23:03:27.849241 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.848968    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a0cfb5-901f-4734-bd40-e6a9f0f53e0f" containerName="revoke-keys"
Oct 29 23:03:27.849241 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.848975    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e574d0f5-9916-4d71-9d2c-8743336447b3" containerName="istio-proxy"
Oct 29 23:03:27.854081 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.854054    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"
Oct 29 23:03:27.867732 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.867703    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"]
Oct 29 23:03:27.871902 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.871875    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"
Oct 29 23:03:27.873432 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.873409    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:03:27.873723 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.873706    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:03:27.882848 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.875398    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"]
Oct 29 23:03:27.892364 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.892199    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"]
Oct 29 23:03:27.923046 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.923004    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c884a500-w878q\" (UID: \"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"
Oct 29 23:03:27.923238 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.923142    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386d9794-ca90-400c-87f9-6a2cb7546982-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b220fd4-lmxtw\" (UID: \"386d9794-ca90-400c-87f9-6a2cb7546982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"
Oct 29 23:03:27.923238 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.923189    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fhvf\" (UniqueName: \"kubernetes.io/projected/386d9794-ca90-400c-87f9-6a2cb7546982-kube-api-access-4fhvf\") pod \"maas-api-revoke-sub-9b220fd4-lmxtw\" (UID: \"386d9794-ca90-400c-87f9-6a2cb7546982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"
Oct 29 23:03:27.923238 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.923225    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386d9794-ca90-400c-87f9-6a2cb7546982-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b220fd4-lmxtw\" (UID: \"386d9794-ca90-400c-87f9-6a2cb7546982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"
Oct 29 23:03:27.923467 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.923251    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m9fh\" (UniqueName: \"kubernetes.io/projected/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-kube-api-access-6m9fh\") pod \"maas-api-revoke-sub-c884a500-w878q\" (UID: \"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"
Oct 29 23:03:27.923467 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:27.923426    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c884a500-w878q\" (UID: \"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"
Oct 29 23:03:28.024789 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:28.024749    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c884a500-w878q\" (UID: \"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"
Oct 29 23:03:28.025010 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:28.024883    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c884a500-w878q\" (UID: \"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"
Oct 29 23:03:28.025010 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:28.024936    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386d9794-ca90-400c-87f9-6a2cb7546982-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b220fd4-lmxtw\" (UID: \"386d9794-ca90-400c-87f9-6a2cb7546982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"
Oct 29 23:03:28.025010 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:28.024968    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fhvf\" (UniqueName: \"kubernetes.io/projected/386d9794-ca90-400c-87f9-6a2cb7546982-kube-api-access-4fhvf\") pod \"maas-api-revoke-sub-9b220fd4-lmxtw\" (UID: \"386d9794-ca90-400c-87f9-6a2cb7546982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"
Oct 29 23:03:28.025010 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:28.024998    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386d9794-ca90-400c-87f9-6a2cb7546982-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b220fd4-lmxtw\" (UID: \"386d9794-ca90-400c-87f9-6a2cb7546982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"
Oct 29 23:03:28.025252 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:28.025021    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m9fh\" (UniqueName: \"kubernetes.io/projected/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-kube-api-access-6m9fh\") pod \"maas-api-revoke-sub-c884a500-w878q\" (UID: \"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"
Oct 29 23:03:28.025619 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:28.025588    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c884a500-w878q\" (UID: \"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"
Oct 29 23:03:28.025956 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:28.025707    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386d9794-ca90-400c-87f9-6a2cb7546982-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b220fd4-lmxtw\" (UID: \"386d9794-ca90-400c-87f9-6a2cb7546982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"
Oct 29 23:03:28.027645 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:28.027616    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386d9794-ca90-400c-87f9-6a2cb7546982-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b220fd4-lmxtw\" (UID: \"386d9794-ca90-400c-87f9-6a2cb7546982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"
Oct 29 23:03:28.027800 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:28.027781    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c884a500-w878q\" (UID: \"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"
Oct 29 23:03:28.044852 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:28.044785    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m9fh\" (UniqueName: \"kubernetes.io/projected/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-kube-api-access-6m9fh\") pod \"maas-api-revoke-sub-c884a500-w878q\" (UID: \"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"
Oct 29 23:03:28.045350 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:28.045323    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fhvf\" (UniqueName: \"kubernetes.io/projected/386d9794-ca90-400c-87f9-6a2cb7546982-kube-api-access-4fhvf\") pod \"maas-api-revoke-sub-9b220fd4-lmxtw\" (UID: \"386d9794-ca90-400c-87f9-6a2cb7546982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"
Oct 29 23:03:28.170015 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:28.169978    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"
Oct 29 23:03:28.194541 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:28.194507    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"
Oct 29 23:03:28.476234 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:28.476196    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"]
Oct 29 23:03:28.476682 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:03:28.476255    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81c8641_eaee_4e4e_8af5_a732e9cb6fcd.slice/crio-76c1e0fc76a20f5223eaac28656aa26e905162c846330e5c0fb8cd8d384851f0 WatchSource:0}: Error finding container 76c1e0fc76a20f5223eaac28656aa26e905162c846330e5c0fb8cd8d384851f0: Status 404 returned error can't find the container with id 76c1e0fc76a20f5223eaac28656aa26e905162c846330e5c0fb8cd8d384851f0
Oct 29 23:03:28.519332 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:28.519286    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"]
Oct 29 23:03:28.520130 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:03:28.520097    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386d9794_ca90_400c_87f9_6a2cb7546982.slice/crio-dcf431419231b105b78634a4d76ae21081586847e66d2dfaddb43f790933f49d WatchSource:0}: Error finding container dcf431419231b105b78634a4d76ae21081586847e66d2dfaddb43f790933f49d: Status 404 returned error can't find the container with id dcf431419231b105b78634a4d76ae21081586847e66d2dfaddb43f790933f49d
Oct 29 23:03:29.080862 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:29.080756    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:03:29.228568 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:29.228532    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="299a03af-cb40-46b1-b42d-8afa20c7cd00" path="/var/lib/kubelet/pods/299a03af-cb40-46b1-b42d-8afa20c7cd00/volumes"
Oct 29 23:03:29.229095 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:29.229071    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="341bcfef-7b8c-4732-a60c-2849614df3b7" path="/var/lib/kubelet/pods/341bcfef-7b8c-4732-a60c-2849614df3b7/volumes"
Oct 29 23:03:29.229602 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:29.229582    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e574d0f5-9916-4d71-9d2c-8743336447b3" path="/var/lib/kubelet/pods/e574d0f5-9916-4d71-9d2c-8743336447b3/volumes"
Oct 29 23:03:29.447689 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:29.447651    4559 generic.go:356] "Generic (PLEG): container finished" podID="f81c8641-eaee-4e4e-8af5-a732e9cb6fcd" containerID="791a8c80fedcda3c64072435609b47000df9dfff328bc2ae2e7d0b8e3f14cd47" exitCode=0
Oct 29 23:03:29.447864 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:29.447740    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q" event={"ID":"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd","Type":"ContainerDied","Data":"791a8c80fedcda3c64072435609b47000df9dfff328bc2ae2e7d0b8e3f14cd47"}
Oct 29 23:03:29.447864 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:29.447832    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q" event={"ID":"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd","Type":"ContainerStarted","Data":"76c1e0fc76a20f5223eaac28656aa26e905162c846330e5c0fb8cd8d384851f0"}
Oct 29 23:03:29.449668 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:29.449640    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw" event={"ID":"386d9794-ca90-400c-87f9-6a2cb7546982","Type":"ContainerStarted","Data":"336135d851a763bed51e21d42919b277e32b50275e12189bf0a1718928b96c2d"}
Oct 29 23:03:29.449821 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:29.449686    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw" event={"ID":"386d9794-ca90-400c-87f9-6a2cb7546982","Type":"ContainerStarted","Data":"dcf431419231b105b78634a4d76ae21081586847e66d2dfaddb43f790933f49d"}
Oct 29 23:03:29.525935 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:29.525870    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw" podStartSLOduration=2.525852085 podStartE2EDuration="2.525852085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:29.524836085 +0000 UTC m=+2948.436279645" watchObservedRunningTime="2026-10-29 23:03:29.525852085 +0000 UTC m=+2948.437295662"
Oct 29 23:03:30.460007 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:30.459968    4559 generic.go:356] "Generic (PLEG): container finished" podID="386d9794-ca90-400c-87f9-6a2cb7546982" containerID="336135d851a763bed51e21d42919b277e32b50275e12189bf0a1718928b96c2d" exitCode=0
Oct 29 23:03:30.460237 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:30.460056    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw" event={"ID":"386d9794-ca90-400c-87f9-6a2cb7546982","Type":"ContainerDied","Data":"336135d851a763bed51e21d42919b277e32b50275e12189bf0a1718928b96c2d"}
Oct 29 23:03:30.553890 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:30.553830    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx" podUID="c7b2c16e-ff49-4b9f-9e75-b0d2c4544532" containerName="payload-pre-processing" containerID="cri-o://c2679dde734fc60ca409d39bda5f9fa652f7453dd299a0e52103a34ee7995826" gracePeriod=25
Oct 29 23:03:30.601997 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:30.601962    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx" podUID="c7b2c16e-ff49-4b9f-9e75-b0d2c4544532" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:30.602332 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:30.602302    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7" podUID="b67ef029-68f6-4d3e-acf1-6637d11c9ae2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:30.628011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:30.627984    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"
Oct 29 23:03:30.658852 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:30.658774    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-maas-api-xks-ca\") pod \"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd\" (UID: \"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd\") "
Oct 29 23:03:30.659062 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:30.658924    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-maas-api-openshift-ca\") pod \"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd\" (UID: \"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd\") "
Oct 29 23:03:30.659062 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:30.659026    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-kube-api-access-6m9fh\" (UniqueName: \"kubernetes.io/projected/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-kube-api-access-6m9fh\") pod \"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd\" (UID: \"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd\") "
Oct 29 23:03:30.659365 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:30.659333    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-maas-api-openshift-ca" pod "f81c8641-eaee-4e4e-8af5-a732e9cb6fcd" (UID: "f81c8641-eaee-4e4e-8af5-a732e9cb6fcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:30.659532 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:30.659509    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:30.661470 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:30.661439    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-kube-api-access-6m9fh" pod "f81c8641-eaee-4e4e-8af5-a732e9cb6fcd" (UID: "f81c8641-eaee-4e4e-8af5-a732e9cb6fcd"). InnerVolumeSpecName "kube-api-access-6m9fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:30.661619 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:30.661522    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-maas-api-xks-ca" pod "f81c8641-eaee-4e4e-8af5-a732e9cb6fcd" (UID: "f81c8641-eaee-4e4e-8af5-a732e9cb6fcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:30.761254 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:30.761147    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:30.761254 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:30.761182    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m9fh\" (UniqueName: \"kubernetes.io/projected/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd-kube-api-access-6m9fh\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:31.475798 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:31.475767    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"
Oct 29 23:03:31.476010 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:31.475804    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q" event={"ID":"f81c8641-eaee-4e4e-8af5-a732e9cb6fcd","Type":"ContainerDied","Data":"76c1e0fc76a20f5223eaac28656aa26e905162c846330e5c0fb8cd8d384851f0"}
Oct 29 23:03:31.476010 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:31.475932    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76c1e0fc76a20f5223eaac28656aa26e905162c846330e5c0fb8cd8d384851f0"
Oct 29 23:03:31.643948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:31.643915    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"
Oct 29 23:03:31.676470 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:31.676439    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386d9794-ca90-400c-87f9-6a2cb7546982-kube-api-access-4fhvf\" (UniqueName: \"kubernetes.io/projected/386d9794-ca90-400c-87f9-6a2cb7546982-kube-api-access-4fhvf\") pod \"386d9794-ca90-400c-87f9-6a2cb7546982\" (UID: \"386d9794-ca90-400c-87f9-6a2cb7546982\") "
Oct 29 23:03:31.676674 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:31.676515    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/386d9794-ca90-400c-87f9-6a2cb7546982-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386d9794-ca90-400c-87f9-6a2cb7546982-maas-api-xks-ca\") pod \"386d9794-ca90-400c-87f9-6a2cb7546982\" (UID: \"386d9794-ca90-400c-87f9-6a2cb7546982\") "
Oct 29 23:03:31.676735 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:31.676674    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386d9794-ca90-400c-87f9-6a2cb7546982-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386d9794-ca90-400c-87f9-6a2cb7546982-maas-api-openshift-ca\") pod \"386d9794-ca90-400c-87f9-6a2cb7546982\" (UID: \"386d9794-ca90-400c-87f9-6a2cb7546982\") "
Oct 29 23:03:31.677168 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:31.677135    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386d9794-ca90-400c-87f9-6a2cb7546982-maas-api-openshift-ca" pod "386d9794-ca90-400c-87f9-6a2cb7546982" (UID: "386d9794-ca90-400c-87f9-6a2cb7546982"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:31.679009 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:31.678977    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386d9794-ca90-400c-87f9-6a2cb7546982-kube-api-access-4fhvf" pod "386d9794-ca90-400c-87f9-6a2cb7546982" (UID: "386d9794-ca90-400c-87f9-6a2cb7546982"). InnerVolumeSpecName "kube-api-access-4fhvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:31.679320 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:31.679298    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/386d9794-ca90-400c-87f9-6a2cb7546982-maas-api-xks-ca" pod "386d9794-ca90-400c-87f9-6a2cb7546982" (UID: "386d9794-ca90-400c-87f9-6a2cb7546982"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:31.778176 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:31.778134    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fhvf\" (UniqueName: \"kubernetes.io/projected/386d9794-ca90-400c-87f9-6a2cb7546982-kube-api-access-4fhvf\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:31.778176 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:31.778178    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386d9794-ca90-400c-87f9-6a2cb7546982-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:31.778404 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:31.778192    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386d9794-ca90-400c-87f9-6a2cb7546982-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:32.486315 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:32.486282    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"
Oct 29 23:03:32.486315 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:32.486305    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw" event={"ID":"386d9794-ca90-400c-87f9-6a2cb7546982","Type":"ContainerDied","Data":"dcf431419231b105b78634a4d76ae21081586847e66d2dfaddb43f790933f49d"}
Oct 29 23:03:32.486607 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:32.486356    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcf431419231b105b78634a4d76ae21081586847e66d2dfaddb43f790933f49d"
Oct 29 23:03:32.769735 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:32.769635    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"]
Oct 29 23:03:32.800780 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:32.800744    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"]
Oct 29 23:03:34.879445 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:34.879393    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main" containerID="cri-o://91d64a1cfcfd9dabeb3fa3e22dd7e3ab6069ff575a7b828c44a5ab24d211c2e9" gracePeriod=45
Oct 29 23:03:35.143989 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.143962    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 23:03:35.220157 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.220120    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-home\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-home\") pod \"6d535825-4bca-4a43-8e4e-4fcae50adffa\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") "
Oct 29 23:03:35.220377 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.220198    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-tmp-dir\") pod \"6d535825-4bca-4a43-8e4e-4fcae50adffa\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") "
Oct 29 23:03:35.220377 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.220310    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d535825-4bca-4a43-8e4e-4fcae50adffa-kube-api-access-pkmzv\" (UniqueName: \"kubernetes.io/projected/6d535825-4bca-4a43-8e4e-4fcae50adffa-kube-api-access-pkmzv\") pod \"6d535825-4bca-4a43-8e4e-4fcae50adffa\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") "
Oct 29 23:03:35.220527 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.220448    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-dshm\") pod \"6d535825-4bca-4a43-8e4e-4fcae50adffa\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") "
Oct 29 23:03:35.220604 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.220551    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-model-cache\") pod \"6d535825-4bca-4a43-8e4e-4fcae50adffa\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") "
Oct 29 23:03:35.220604 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.220583    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d535825-4bca-4a43-8e4e-4fcae50adffa-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d535825-4bca-4a43-8e4e-4fcae50adffa-tls-certs\") pod \"6d535825-4bca-4a43-8e4e-4fcae50adffa\" (UID: \"6d535825-4bca-4a43-8e4e-4fcae50adffa\") "
Oct 29 23:03:35.222365 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.222299    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-model-cache" pod "6d535825-4bca-4a43-8e4e-4fcae50adffa" (UID: "6d535825-4bca-4a43-8e4e-4fcae50adffa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:35.222597 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.222546    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-tmp-dir" pod "6d535825-4bca-4a43-8e4e-4fcae50adffa" (UID: "6d535825-4bca-4a43-8e4e-4fcae50adffa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:35.222825 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.222776    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-home" pod "6d535825-4bca-4a43-8e4e-4fcae50adffa" (UID: "6d535825-4bca-4a43-8e4e-4fcae50adffa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:35.224362 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.224319    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d535825-4bca-4a43-8e4e-4fcae50adffa-kube-api-access-pkmzv" pod "6d535825-4bca-4a43-8e4e-4fcae50adffa" (UID: "6d535825-4bca-4a43-8e4e-4fcae50adffa"). InnerVolumeSpecName "kube-api-access-pkmzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:35.224493 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.224444    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d535825-4bca-4a43-8e4e-4fcae50adffa-tls-certs" pod "6d535825-4bca-4a43-8e4e-4fcae50adffa" (UID: "6d535825-4bca-4a43-8e4e-4fcae50adffa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:35.231158 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.231120    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-dshm" pod "6d535825-4bca-4a43-8e4e-4fcae50adffa" (UID: "6d535825-4bca-4a43-8e4e-4fcae50adffa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:35.322348 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.322312    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkmzv\" (UniqueName: \"kubernetes.io/projected/6d535825-4bca-4a43-8e4e-4fcae50adffa-kube-api-access-pkmzv\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.322348 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.322344    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-dshm\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.322348 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.322354    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-model-cache\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.322592 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.322363    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d535825-4bca-4a43-8e4e-4fcae50adffa-tls-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.322592 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.322372    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-home\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.322592 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.322379    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d535825-4bca-4a43-8e4e-4fcae50adffa-tmp-dir\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.510072 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.509963    4559 generic.go:356] "Generic (PLEG): container finished" podID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerID="91d64a1cfcfd9dabeb3fa3e22dd7e3ab6069ff575a7b828c44a5ab24d211c2e9" exitCode=0
Oct 29 23:03:35.510072 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.510027    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"
Oct 29 23:03:35.510072 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.510055    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" event={"ID":"6d535825-4bca-4a43-8e4e-4fcae50adffa","Type":"ContainerDied","Data":"91d64a1cfcfd9dabeb3fa3e22dd7e3ab6069ff575a7b828c44a5ab24d211c2e9"}
Oct 29 23:03:35.510370 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.510091    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt" event={"ID":"6d535825-4bca-4a43-8e4e-4fcae50adffa","Type":"ContainerDied","Data":"0829983355d86de87e6037fb723811e8f4473148dba74ebb3605c71048b014d4"}
Oct 29 23:03:35.510370 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.510110    4559 scope.go:122] "RemoveContainer" containerID="91d64a1cfcfd9dabeb3fa3e22dd7e3ab6069ff575a7b828c44a5ab24d211c2e9"
Oct 29 23:03:35.521890 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.521854    4559 scope.go:122] "RemoveContainer" containerID="91d64a1cfcfd9dabeb3fa3e22dd7e3ab6069ff575a7b828c44a5ab24d211c2e9"
Oct 29 23:03:35.522236 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:35.522212    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d64a1cfcfd9dabeb3fa3e22dd7e3ab6069ff575a7b828c44a5ab24d211c2e9\": container with ID starting with 91d64a1cfcfd9dabeb3fa3e22dd7e3ab6069ff575a7b828c44a5ab24d211c2e9 not found: ID does not exist" containerID="91d64a1cfcfd9dabeb3fa3e22dd7e3ab6069ff575a7b828c44a5ab24d211c2e9"
Oct 29 23:03:35.522322 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.522250    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d64a1cfcfd9dabeb3fa3e22dd7e3ab6069ff575a7b828c44a5ab24d211c2e9"} err="failed to get container status \"91d64a1cfcfd9dabeb3fa3e22dd7e3ab6069ff575a7b828c44a5ab24d211c2e9\": rpc error: code = NotFound desc = could not find container \"91d64a1cfcfd9dabeb3fa3e22dd7e3ab6069ff575a7b828c44a5ab24d211c2e9\": container with ID starting with 91d64a1cfcfd9dabeb3fa3e22dd7e3ab6069ff575a7b828c44a5ab24d211c2e9 not found: ID does not exist"
Oct 29 23:03:35.562244 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.562206    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"]
Oct 29 23:03:35.574776 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:35.574744    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/simulator-w2-2ba10a-kserve-7f98c9f794-rb5rt"]
Oct 29 23:03:36.662529 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:36.662486    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"]
Oct 29 23:03:36.663044 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:36.662784    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6" podUID="db95228d-b0d3-41df-966d-770f4f338a72" containerName="payload-processing" containerID="cri-o://f0ce699d1f59de4a54ac032a6076203a54c787e633cbe3aa08a0be790f968b5b" gracePeriod=30
Oct 29 23:03:36.695063 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:36.695021    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"]
Oct 29 23:03:36.695350 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:36.695294    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml" podUID="a0fc466b-5f14-4f21-bba8-d8be8f30323f" containerName="payload-pre-processing" containerID="cri-o://e4ba5696da3753449dfaac8c92ad706d1ddc41ba347c7a6a76ed6012d1451326" gracePeriod=30
Oct 29 23:03:36.989399 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:36.989366    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"
Oct 29 23:03:37.061833 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.061603    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db95228d-b0d3-41df-966d-770f4f338a72-kube-api-access-5h7qk\" (UniqueName: \"kubernetes.io/projected/db95228d-b0d3-41df-966d-770f4f338a72-kube-api-access-5h7qk\") pod \"db95228d-b0d3-41df-966d-770f4f338a72\" (UID: \"db95228d-b0d3-41df-966d-770f4f338a72\") "
Oct 29 23:03:37.061833 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.061732    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db95228d-b0d3-41df-966d-770f4f338a72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db95228d-b0d3-41df-966d-770f4f338a72-plugins-config-volume\") pod \"db95228d-b0d3-41df-966d-770f4f338a72\" (UID: \"db95228d-b0d3-41df-966d-770f4f338a72\") "
Oct 29 23:03:37.091071 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.087042    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db95228d-b0d3-41df-966d-770f4f338a72-plugins-config-volume" pod "db95228d-b0d3-41df-966d-770f4f338a72" (UID: "db95228d-b0d3-41df-966d-770f4f338a72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:37.093069 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.092931    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db95228d-b0d3-41df-966d-770f4f338a72-kube-api-access-5h7qk" pod "db95228d-b0d3-41df-966d-770f4f338a72" (UID: "db95228d-b0d3-41df-966d-770f4f338a72"). InnerVolumeSpecName "kube-api-access-5h7qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:37.098995 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.098667    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"
Oct 29 23:03:37.163173 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.163136    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0fc466b-5f14-4f21-bba8-d8be8f30323f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fc466b-5f14-4f21-bba8-d8be8f30323f-plugins-config-volume\") pod \"a0fc466b-5f14-4f21-bba8-d8be8f30323f\" (UID: \"a0fc466b-5f14-4f21-bba8-d8be8f30323f\") "
Oct 29 23:03:37.163395 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.163249    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0fc466b-5f14-4f21-bba8-d8be8f30323f-kube-api-access-bxd6k\" (UniqueName: \"kubernetes.io/projected/a0fc466b-5f14-4f21-bba8-d8be8f30323f-kube-api-access-bxd6k\") pod \"a0fc466b-5f14-4f21-bba8-d8be8f30323f\" (UID: \"a0fc466b-5f14-4f21-bba8-d8be8f30323f\") "
Oct 29 23:03:37.163624 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.163585    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0fc466b-5f14-4f21-bba8-d8be8f30323f-plugins-config-volume" pod "a0fc466b-5f14-4f21-bba8-d8be8f30323f" (UID: "a0fc466b-5f14-4f21-bba8-d8be8f30323f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:37.163757 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.163735    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h7qk\" (UniqueName: \"kubernetes.io/projected/db95228d-b0d3-41df-966d-770f4f338a72-kube-api-access-5h7qk\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:37.163841 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.163761    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fc466b-5f14-4f21-bba8-d8be8f30323f-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:37.163841 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.163774    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db95228d-b0d3-41df-966d-770f4f338a72-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:37.165670 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.165636    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0fc466b-5f14-4f21-bba8-d8be8f30323f-kube-api-access-bxd6k" pod "a0fc466b-5f14-4f21-bba8-d8be8f30323f" (UID: "a0fc466b-5f14-4f21-bba8-d8be8f30323f"). InnerVolumeSpecName "kube-api-access-bxd6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:37.228386 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.228296    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" path="/var/lib/kubelet/pods/6d535825-4bca-4a43-8e4e-4fcae50adffa/volumes"
Oct 29 23:03:37.264689 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.264653    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxd6k\" (UniqueName: \"kubernetes.io/projected/a0fc466b-5f14-4f21-bba8-d8be8f30323f-kube-api-access-bxd6k\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:37.525527 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.525434    4559 generic.go:356] "Generic (PLEG): container finished" podID="a0fc466b-5f14-4f21-bba8-d8be8f30323f" containerID="e4ba5696da3753449dfaac8c92ad706d1ddc41ba347c7a6a76ed6012d1451326" exitCode=0
Oct 29 23:03:37.525527 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.525483    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"
Oct 29 23:03:37.525776 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.525522    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml" event={"ID":"a0fc466b-5f14-4f21-bba8-d8be8f30323f","Type":"ContainerDied","Data":"e4ba5696da3753449dfaac8c92ad706d1ddc41ba347c7a6a76ed6012d1451326"}
Oct 29 23:03:37.525776 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.525581    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml" event={"ID":"a0fc466b-5f14-4f21-bba8-d8be8f30323f","Type":"ContainerDied","Data":"61712457b1e2312a688dee9f3f0f57330302fe5843bd3ccf22e7759db453fe36"}
Oct 29 23:03:37.525776 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.525607    4559 scope.go:122] "RemoveContainer" containerID="e4ba5696da3753449dfaac8c92ad706d1ddc41ba347c7a6a76ed6012d1451326"
Oct 29 23:03:37.527610 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.527531    4559 generic.go:356] "Generic (PLEG): container finished" podID="db95228d-b0d3-41df-966d-770f4f338a72" containerID="f0ce699d1f59de4a54ac032a6076203a54c787e633cbe3aa08a0be790f968b5b" exitCode=0
Oct 29 23:03:37.527610 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.527574    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6" event={"ID":"db95228d-b0d3-41df-966d-770f4f338a72","Type":"ContainerDied","Data":"f0ce699d1f59de4a54ac032a6076203a54c787e633cbe3aa08a0be790f968b5b"}
Oct 29 23:03:37.527610 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.527578    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"
Oct 29 23:03:37.527872 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.527619    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6" event={"ID":"db95228d-b0d3-41df-966d-770f4f338a72","Type":"ContainerDied","Data":"883e9c661583d603ec99e9077172da786ec2f3dd71a2540aaab1b4f9efbd06a9"}
Oct 29 23:03:37.537941 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.537918    4559 scope.go:122] "RemoveContainer" containerID="e4ba5696da3753449dfaac8c92ad706d1ddc41ba347c7a6a76ed6012d1451326"
Oct 29 23:03:37.538309 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:37.538285    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ba5696da3753449dfaac8c92ad706d1ddc41ba347c7a6a76ed6012d1451326\": container with ID starting with e4ba5696da3753449dfaac8c92ad706d1ddc41ba347c7a6a76ed6012d1451326 not found: ID does not exist" containerID="e4ba5696da3753449dfaac8c92ad706d1ddc41ba347c7a6a76ed6012d1451326"
Oct 29 23:03:37.538405 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.538317    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ba5696da3753449dfaac8c92ad706d1ddc41ba347c7a6a76ed6012d1451326"} err="failed to get container status \"e4ba5696da3753449dfaac8c92ad706d1ddc41ba347c7a6a76ed6012d1451326\": rpc error: code = NotFound desc = could not find container \"e4ba5696da3753449dfaac8c92ad706d1ddc41ba347c7a6a76ed6012d1451326\": container with ID starting with e4ba5696da3753449dfaac8c92ad706d1ddc41ba347c7a6a76ed6012d1451326 not found: ID does not exist"
Oct 29 23:03:37.538405 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.538336    4559 scope.go:122] "RemoveContainer" containerID="f0ce699d1f59de4a54ac032a6076203a54c787e633cbe3aa08a0be790f968b5b"
Oct 29 23:03:37.548952 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.548928    4559 scope.go:122] "RemoveContainer" containerID="f0ce699d1f59de4a54ac032a6076203a54c787e633cbe3aa08a0be790f968b5b"
Oct 29 23:03:37.549312 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:37.549289    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ce699d1f59de4a54ac032a6076203a54c787e633cbe3aa08a0be790f968b5b\": container with ID starting with f0ce699d1f59de4a54ac032a6076203a54c787e633cbe3aa08a0be790f968b5b not found: ID does not exist" containerID="f0ce699d1f59de4a54ac032a6076203a54c787e633cbe3aa08a0be790f968b5b"
Oct 29 23:03:37.549393 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.549320    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ce699d1f59de4a54ac032a6076203a54c787e633cbe3aa08a0be790f968b5b"} err="failed to get container status \"f0ce699d1f59de4a54ac032a6076203a54c787e633cbe3aa08a0be790f968b5b\": rpc error: code = NotFound desc = could not find container \"f0ce699d1f59de4a54ac032a6076203a54c787e633cbe3aa08a0be790f968b5b\": container with ID starting with f0ce699d1f59de4a54ac032a6076203a54c787e633cbe3aa08a0be790f968b5b not found: ID does not exist"
Oct 29 23:03:37.578460 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.578425    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"]
Oct 29 23:03:37.590462 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.590426    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-80248b-5c58666f59-csjml"]
Oct 29 23:03:37.617071 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.617035    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"]
Oct 29 23:03:37.621637 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.621601    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-80248b-5dfc48b658-bwlk6"]
Oct 29 23:03:37.770311 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.770255    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh" podUID="7728dea4-21ee-478e-8fcc-c4632e1b350a" containerName="payload-processing" containerID="cri-o://25e990b39d274eaf0eedb28be6d181b7ba8043740fa224f2555de6b4c93207e8" gracePeriod=25
Oct 29 23:03:37.801478 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:37.801369    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n" podUID="2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0" containerName="payload-pre-processing" containerID="cri-o://8ffc843b4979e16d3734b097d84e9798f3d086242c47d4024e5b390806845ff3" gracePeriod=25
Oct 29 23:03:39.228209 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:39.228171    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0fc466b-5f14-4f21-bba8-d8be8f30323f" path="/var/lib/kubelet/pods/a0fc466b-5f14-4f21-bba8-d8be8f30323f/volumes"
Oct 29 23:03:39.228742 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:39.228714    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db95228d-b0d3-41df-966d-770f4f338a72" path="/var/lib/kubelet/pods/db95228d-b0d3-41df-966d-770f4f338a72/volumes"
Oct 29 23:03:40.602151 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:40.602113    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7" podUID="b67ef029-68f6-4d3e-acf1-6637d11c9ae2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:40.602577 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:40.602246    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx" podUID="c7b2c16e-ff49-4b9f-9e75-b0d2c4544532" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:41.182607 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:41.182577    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"
Oct 29 23:03:41.314552 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:41.314518    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67ef029-68f6-4d3e-acf1-6637d11c9ae2-kube-api-access-4vjdd\" (UniqueName: \"kubernetes.io/projected/b67ef029-68f6-4d3e-acf1-6637d11c9ae2-kube-api-access-4vjdd\") pod \"b67ef029-68f6-4d3e-acf1-6637d11c9ae2\" (UID: \"b67ef029-68f6-4d3e-acf1-6637d11c9ae2\") "
Oct 29 23:03:41.314770 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:41.314580    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b67ef029-68f6-4d3e-acf1-6637d11c9ae2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b67ef029-68f6-4d3e-acf1-6637d11c9ae2-plugins-config-volume\") pod \"b67ef029-68f6-4d3e-acf1-6637d11c9ae2\" (UID: \"b67ef029-68f6-4d3e-acf1-6637d11c9ae2\") "
Oct 29 23:03:41.315049 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:41.315022    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b67ef029-68f6-4d3e-acf1-6637d11c9ae2-plugins-config-volume" pod "b67ef029-68f6-4d3e-acf1-6637d11c9ae2" (UID: "b67ef029-68f6-4d3e-acf1-6637d11c9ae2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:41.316958 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:41.316929    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67ef029-68f6-4d3e-acf1-6637d11c9ae2-kube-api-access-4vjdd" pod "b67ef029-68f6-4d3e-acf1-6637d11c9ae2" (UID: "b67ef029-68f6-4d3e-acf1-6637d11c9ae2"). InnerVolumeSpecName "kube-api-access-4vjdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:41.416539 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:41.416450    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vjdd\" (UniqueName: \"kubernetes.io/projected/b67ef029-68f6-4d3e-acf1-6637d11c9ae2-kube-api-access-4vjdd\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:41.416539 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:41.416484    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b67ef029-68f6-4d3e-acf1-6637d11c9ae2-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:41.565244 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:41.565204    4559 generic.go:356] "Generic (PLEG): container finished" podID="b67ef029-68f6-4d3e-acf1-6637d11c9ae2" containerID="b083dc8de6a296a7eacd128265a4e015bcac2836569d28d0a70be40a3ba29603" exitCode=0
Oct 29 23:03:41.565427 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:41.565263    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7" event={"ID":"b67ef029-68f6-4d3e-acf1-6637d11c9ae2","Type":"ContainerDied","Data":"b083dc8de6a296a7eacd128265a4e015bcac2836569d28d0a70be40a3ba29603"}
Oct 29 23:03:41.565427 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:41.565317    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7" event={"ID":"b67ef029-68f6-4d3e-acf1-6637d11c9ae2","Type":"ContainerDied","Data":"fe97ac6a16ccc138a56319125c4cb5ad51baeac65567218f2b43f92e2483a164"}
Oct 29 23:03:41.565427 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:41.565264    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"
Oct 29 23:03:41.565427 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:41.565344    4559 scope.go:122] "RemoveContainer" containerID="b083dc8de6a296a7eacd128265a4e015bcac2836569d28d0a70be40a3ba29603"
Oct 29 23:03:41.575602 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:41.575578    4559 scope.go:122] "RemoveContainer" containerID="b083dc8de6a296a7eacd128265a4e015bcac2836569d28d0a70be40a3ba29603"
Oct 29 23:03:41.576150 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:41.576103    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b083dc8de6a296a7eacd128265a4e015bcac2836569d28d0a70be40a3ba29603\": container with ID starting with b083dc8de6a296a7eacd128265a4e015bcac2836569d28d0a70be40a3ba29603 not found: ID does not exist" containerID="b083dc8de6a296a7eacd128265a4e015bcac2836569d28d0a70be40a3ba29603"
Oct 29 23:03:41.576295 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:41.576151    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b083dc8de6a296a7eacd128265a4e015bcac2836569d28d0a70be40a3ba29603"} err="failed to get container status \"b083dc8de6a296a7eacd128265a4e015bcac2836569d28d0a70be40a3ba29603\": rpc error: code = NotFound desc = could not find container \"b083dc8de6a296a7eacd128265a4e015bcac2836569d28d0a70be40a3ba29603\": container with ID starting with b083dc8de6a296a7eacd128265a4e015bcac2836569d28d0a70be40a3ba29603 not found: ID does not exist"
Oct 29 23:03:41.618274 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:41.618237    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"]
Oct 29 23:03:41.630387 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:41.630352    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2e046a-74bbd64fc9-7nrm7"]
Oct 29 23:03:43.227328 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:43.227294    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b67ef029-68f6-4d3e-acf1-6637d11c9ae2" path="/var/lib/kubelet/pods/b67ef029-68f6-4d3e-acf1-6637d11c9ae2/volumes"
Oct 29 23:03:43.694388 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:43.694345    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:03:44.263147 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.263104    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"]
Oct 29 23:03:44.263729 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.263371    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs" podUID="6f1b07f5-c3bd-4382-9318-2234d4e9093e" containerName="istio-proxy" containerID="cri-o://64b6131ecbb0b458ec41c28aa33d1bafb771bef511c245a8fa796e68001e8c5d" gracePeriod=30
Oct 29 23:03:44.490890 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.490850    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:03:44.491110 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.490923    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs" podUID="6f1b07f5-c3bd-4382-9318-2234d4e9093e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:03:44.707184 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.707140    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"]
Oct 29 23:03:44.713671 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.713637    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-6fc6bd-4qs4q"]
Oct 29 23:03:44.788042 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788006    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"]
Oct 29 23:03:44.788611 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788592    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a0fc466b-5f14-4f21-bba8-d8be8f30323f" containerName="payload-pre-processing"
Oct 29 23:03:44.788611 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788609    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0fc466b-5f14-4f21-bba8-d8be8f30323f" containerName="payload-pre-processing"
Oct 29 23:03:44.788781 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788623    4559 cpu_manager.go:414] "containerMap: removing container" podUID="db95228d-b0d3-41df-966d-770f4f338a72" containerName="payload-processing"
Oct 29 23:03:44.788781 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788633    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db95228d-b0d3-41df-966d-770f4f338a72" containerName="payload-processing"
Oct 29 23:03:44.788781 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788657    4559 cpu_manager.go:414] "containerMap: removing container" podUID="f81c8641-eaee-4e4e-8af5-a732e9cb6fcd" containerName="revoke-keys"
Oct 29 23:03:44.788781 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788662    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f81c8641-eaee-4e4e-8af5-a732e9cb6fcd" containerName="revoke-keys"
Oct 29 23:03:44.788781 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788673    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main"
Oct 29 23:03:44.788781 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788678    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main"
Oct 29 23:03:44.788781 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788687    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b67ef029-68f6-4d3e-acf1-6637d11c9ae2" containerName="payload-pre-processing"
Oct 29 23:03:44.788781 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788692    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b67ef029-68f6-4d3e-acf1-6637d11c9ae2" containerName="payload-pre-processing"
Oct 29 23:03:44.788781 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788700    4559 cpu_manager.go:414] "containerMap: removing container" podUID="386d9794-ca90-400c-87f9-6a2cb7546982" containerName="revoke-keys"
Oct 29 23:03:44.788781 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788705    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386d9794-ca90-400c-87f9-6a2cb7546982" containerName="revoke-keys"
Oct 29 23:03:44.788781 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788778    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0fc466b-5f14-4f21-bba8-d8be8f30323f" containerName="payload-pre-processing"
Oct 29 23:03:44.789182 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788789    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="db95228d-b0d3-41df-966d-770f4f338a72" containerName="payload-processing"
Oct 29 23:03:44.789182 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788798    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="f81c8641-eaee-4e4e-8af5-a732e9cb6fcd" containerName="revoke-keys"
Oct 29 23:03:44.789182 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788821    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d535825-4bca-4a43-8e4e-4fcae50adffa" containerName="main"
Oct 29 23:03:44.789182 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788830    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b67ef029-68f6-4d3e-acf1-6637d11c9ae2" containerName="payload-pre-processing"
Oct 29 23:03:44.789182 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.788836    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="386d9794-ca90-400c-87f9-6a2cb7546982" containerName="revoke-keys"
Oct 29 23:03:44.793590 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.793572    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"
Oct 29 23:03:44.812997 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.812953    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"]
Oct 29 23:03:44.902834 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.902104    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:03:44.902834 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.902297    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:03:44.968432 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.968329    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c90510-e767-4379-95c6-d1ea52e7180f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk\" (UID: \"63c90510-e767-4379-95c6-d1ea52e7180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"
Oct 29 23:03:44.968432 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.968384    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c90510-e767-4379-95c6-d1ea52e7180f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk\" (UID: \"63c90510-e767-4379-95c6-d1ea52e7180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"
Oct 29 23:03:44.968670 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:44.968444    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmnz7\" (UniqueName: \"kubernetes.io/projected/63c90510-e767-4379-95c6-d1ea52e7180f-kube-api-access-lmnz7\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk\" (UID: \"63c90510-e767-4379-95c6-d1ea52e7180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"
Oct 29 23:03:45.074832 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:45.069517    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c90510-e767-4379-95c6-d1ea52e7180f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk\" (UID: \"63c90510-e767-4379-95c6-d1ea52e7180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"
Oct 29 23:03:45.074832 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:45.069605    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c90510-e767-4379-95c6-d1ea52e7180f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk\" (UID: \"63c90510-e767-4379-95c6-d1ea52e7180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"
Oct 29 23:03:45.074832 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:45.069636    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmnz7\" (UniqueName: \"kubernetes.io/projected/63c90510-e767-4379-95c6-d1ea52e7180f-kube-api-access-lmnz7\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk\" (UID: \"63c90510-e767-4379-95c6-d1ea52e7180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"
Oct 29 23:03:45.074832 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:45.071237    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c90510-e767-4379-95c6-d1ea52e7180f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk\" (UID: \"63c90510-e767-4379-95c6-d1ea52e7180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"
Oct 29 23:03:45.093843 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:45.087409    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c90510-e767-4379-95c6-d1ea52e7180f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk\" (UID: \"63c90510-e767-4379-95c6-d1ea52e7180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"
Oct 29 23:03:45.094039 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:45.093892    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmnz7\" (UniqueName: \"kubernetes.io/projected/63c90510-e767-4379-95c6-d1ea52e7180f-kube-api-access-lmnz7\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk\" (UID: \"63c90510-e767-4379-95c6-d1ea52e7180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"
Oct 29 23:03:45.132023 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:45.120411    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"
Oct 29 23:03:45.233533 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:45.233444    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a0cfb5-901f-4734-bd40-e6a9f0f53e0f" path="/var/lib/kubelet/pods/04a0cfb5-901f-4734-bd40-e6a9f0f53e0f/volumes"
Oct 29 23:03:45.503985 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:45.503952    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"]
Oct 29 23:03:45.505362 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:03:45.505320    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c90510_e767_4379_95c6_d1ea52e7180f.slice/crio-690233b9f00aea042db32d98e10718948392cb70a78eb9ffed4675dc64b3e489 WatchSource:0}: Error finding container 690233b9f00aea042db32d98e10718948392cb70a78eb9ffed4675dc64b3e489: Status 404 returned error can't find the container with id 690233b9f00aea042db32d98e10718948392cb70a78eb9ffed4675dc64b3e489
Oct 29 23:03:45.596083 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:45.596043    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk" event={"ID":"63c90510-e767-4379-95c6-d1ea52e7180f","Type":"ContainerStarted","Data":"690233b9f00aea042db32d98e10718948392cb70a78eb9ffed4675dc64b3e489"}
Oct 29 23:03:46.609854 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:46.609788    4559 generic.go:356] "Generic (PLEG): container finished" podID="63c90510-e767-4379-95c6-d1ea52e7180f" containerID="b8c97c6100c84e43105c3b84e4f0d97229e91c5059ee4429d4123eb503a9f020" exitCode=0
Oct 29 23:03:46.609854 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:46.609853    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk" event={"ID":"63c90510-e767-4379-95c6-d1ea52e7180f","Type":"ContainerDied","Data":"b8c97c6100c84e43105c3b84e4f0d97229e91c5059ee4429d4123eb503a9f020"}
Oct 29 23:03:47.332488 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:47.332454    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh" podUID="7728dea4-21ee-478e-8fcc-c4632e1b350a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:47.332696 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:47.332453    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n" podUID="2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:47.452403 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:47.452365    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"]
Oct 29 23:03:47.466320 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:47.466285    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2ba10a-r92zk"]
Oct 29 23:03:47.776433 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:47.776402    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"
Oct 29 23:03:47.818783 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:47.818364    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c90510-e767-4379-95c6-d1ea52e7180f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c90510-e767-4379-95c6-d1ea52e7180f-maas-api-xks-ca\") pod \"63c90510-e767-4379-95c6-d1ea52e7180f\" (UID: \"63c90510-e767-4379-95c6-d1ea52e7180f\") "
Oct 29 23:03:47.818783 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:47.818437    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c90510-e767-4379-95c6-d1ea52e7180f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c90510-e767-4379-95c6-d1ea52e7180f-maas-api-openshift-ca\") pod \"63c90510-e767-4379-95c6-d1ea52e7180f\" (UID: \"63c90510-e767-4379-95c6-d1ea52e7180f\") "
Oct 29 23:03:47.818783 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:47.818671    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c90510-e767-4379-95c6-d1ea52e7180f-kube-api-access-lmnz7\" (UniqueName: \"kubernetes.io/projected/63c90510-e767-4379-95c6-d1ea52e7180f-kube-api-access-lmnz7\") pod \"63c90510-e767-4379-95c6-d1ea52e7180f\" (UID: \"63c90510-e767-4379-95c6-d1ea52e7180f\") "
Oct 29 23:03:47.819299 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:47.819255    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c90510-e767-4379-95c6-d1ea52e7180f-maas-api-openshift-ca" pod "63c90510-e767-4379-95c6-d1ea52e7180f" (UID: "63c90510-e767-4379-95c6-d1ea52e7180f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:47.822200 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:47.822171    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c90510-e767-4379-95c6-d1ea52e7180f-maas-api-xks-ca" pod "63c90510-e767-4379-95c6-d1ea52e7180f" (UID: "63c90510-e767-4379-95c6-d1ea52e7180f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:47.822364 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:47.822213    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c90510-e767-4379-95c6-d1ea52e7180f-kube-api-access-lmnz7" pod "63c90510-e767-4379-95c6-d1ea52e7180f" (UID: "63c90510-e767-4379-95c6-d1ea52e7180f"). InnerVolumeSpecName "kube-api-access-lmnz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:47.920643 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:47.920603    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c90510-e767-4379-95c6-d1ea52e7180f-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:47.920643 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:47.920640    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c90510-e767-4379-95c6-d1ea52e7180f-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:47.920643 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:47.920651    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmnz7\" (UniqueName: \"kubernetes.io/projected/63c90510-e767-4379-95c6-d1ea52e7180f-kube-api-access-lmnz7\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:48.628930 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:48.628886    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk" event={"ID":"63c90510-e767-4379-95c6-d1ea52e7180f","Type":"ContainerDied","Data":"690233b9f00aea042db32d98e10718948392cb70a78eb9ffed4675dc64b3e489"}
Oct 29 23:03:48.628930 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:48.628928    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"
Oct 29 23:03:48.629225 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:48.628949    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="690233b9f00aea042db32d98e10718948392cb70a78eb9ffed4675dc64b3e489"
Oct 29 23:03:49.227846 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.227798    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16a423d7-c2f3-4fc3-809b-debb115ff543" path="/var/lib/kubelet/pods/16a423d7-c2f3-4fc3-809b-debb115ff543/volumes"
Oct 29 23:03:49.542768 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.542740    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 23:03:49.638951 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.638914    4559 generic.go:356] "Generic (PLEG): container finished" podID="6f1b07f5-c3bd-4382-9318-2234d4e9093e" containerID="64b6131ecbb0b458ec41c28aa33d1bafb771bef511c245a8fa796e68001e8c5d" exitCode=0
Oct 29 23:03:49.639143 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.638961    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"
Oct 29 23:03:49.639143 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.638991    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs" event={"ID":"6f1b07f5-c3bd-4382-9318-2234d4e9093e","Type":"ContainerDied","Data":"64b6131ecbb0b458ec41c28aa33d1bafb771bef511c245a8fa796e68001e8c5d"}
Oct 29 23:03:49.639143 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.639038    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs" event={"ID":"6f1b07f5-c3bd-4382-9318-2234d4e9093e","Type":"ContainerDied","Data":"125ee54355607d87e323cd5eafa0268e345c876f04db1ceb6457123d927736b7"}
Oct 29 23:03:49.639143 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.639058    4559 scope.go:122] "RemoveContainer" containerID="64b6131ecbb0b458ec41c28aa33d1bafb771bef511c245a8fa796e68001e8c5d"
Oct 29 23:03:49.639143 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.638925    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-workload-socket\") pod \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") "
Oct 29 23:03:49.639143 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.639143    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-credential-socket\") pod \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") "
Oct 29 23:03:49.639458 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.639158    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-workload-socket" pod "6f1b07f5-c3bd-4382-9318-2234d4e9093e" (UID: "6f1b07f5-c3bd-4382-9318-2234d4e9093e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:49.639458 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.639194    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f1b07f5-c3bd-4382-9318-2234d4e9093e-kube-api-access-dgw2f\" (UniqueName: \"kubernetes.io/projected/6f1b07f5-c3bd-4382-9318-2234d4e9093e-kube-api-access-dgw2f\") pod \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") "
Oct 29 23:03:49.639458 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.639268    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-workload-certs\") pod \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") "
Oct 29 23:03:49.639458 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.639302    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istiod-ca-cert\") pod \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") "
Oct 29 23:03:49.639458 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.639344    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-podinfo\") pod \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") "
Oct 29 23:03:49.639458 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.639428    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-data\") pod \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") "
Oct 29 23:03:49.639458 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.639456    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-envoy\") pod \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") "
Oct 29 23:03:49.639888 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.639486    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-token\" (UniqueName: \"kubernetes.io/projected/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-token\") pod \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\" (UID: \"6f1b07f5-c3bd-4382-9318-2234d4e9093e\") "
Oct 29 23:03:49.639888 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.639510    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-credential-socket" pod "6f1b07f5-c3bd-4382-9318-2234d4e9093e" (UID: "6f1b07f5-c3bd-4382-9318-2234d4e9093e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:49.639888 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.639563    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-workload-certs" pod "6f1b07f5-c3bd-4382-9318-2234d4e9093e" (UID: "6f1b07f5-c3bd-4382-9318-2234d4e9093e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:49.639888 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.639699    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istiod-ca-cert" pod "6f1b07f5-c3bd-4382-9318-2234d4e9093e" (UID: "6f1b07f5-c3bd-4382-9318-2234d4e9093e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:49.640145 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.640017    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-data" pod "6f1b07f5-c3bd-4382-9318-2234d4e9093e" (UID: "6f1b07f5-c3bd-4382-9318-2234d4e9093e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:49.640217 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.640180    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:49.640217 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.640204    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:49.640499 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.640219    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:49.640499 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.640232    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:49.640499 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.640245    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:49.642049 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.642011    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-podinfo" pod "6f1b07f5-c3bd-4382-9318-2234d4e9093e" (UID: "6f1b07f5-c3bd-4382-9318-2234d4e9093e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:49.642049 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.642009    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-envoy" pod "6f1b07f5-c3bd-4382-9318-2234d4e9093e" (UID: "6f1b07f5-c3bd-4382-9318-2234d4e9093e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:49.642321 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.642300    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-token" pod "6f1b07f5-c3bd-4382-9318-2234d4e9093e" (UID: "6f1b07f5-c3bd-4382-9318-2234d4e9093e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:49.642390 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.642369    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1b07f5-c3bd-4382-9318-2234d4e9093e-kube-api-access-dgw2f" pod "6f1b07f5-c3bd-4382-9318-2234d4e9093e" (UID: "6f1b07f5-c3bd-4382-9318-2234d4e9093e"). InnerVolumeSpecName "kube-api-access-dgw2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:49.665925 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.665899    4559 scope.go:122] "RemoveContainer" containerID="be840790c395da14e994e5eed7278ff4ab0f9d97d887505937f7ee3cc9f1e3ff"
Oct 29 23:03:49.676235 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.676212    4559 scope.go:122] "RemoveContainer" containerID="64b6131ecbb0b458ec41c28aa33d1bafb771bef511c245a8fa796e68001e8c5d"
Oct 29 23:03:49.676551 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:49.676526    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b6131ecbb0b458ec41c28aa33d1bafb771bef511c245a8fa796e68001e8c5d\": container with ID starting with 64b6131ecbb0b458ec41c28aa33d1bafb771bef511c245a8fa796e68001e8c5d not found: ID does not exist" containerID="64b6131ecbb0b458ec41c28aa33d1bafb771bef511c245a8fa796e68001e8c5d"
Oct 29 23:03:49.676661 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.676557    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b6131ecbb0b458ec41c28aa33d1bafb771bef511c245a8fa796e68001e8c5d"} err="failed to get container status \"64b6131ecbb0b458ec41c28aa33d1bafb771bef511c245a8fa796e68001e8c5d\": rpc error: code = NotFound desc = could not find container \"64b6131ecbb0b458ec41c28aa33d1bafb771bef511c245a8fa796e68001e8c5d\": container with ID starting with 64b6131ecbb0b458ec41c28aa33d1bafb771bef511c245a8fa796e68001e8c5d not found: ID does not exist"
Oct 29 23:03:49.676661 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.676579    4559 scope.go:122] "RemoveContainer" containerID="be840790c395da14e994e5eed7278ff4ab0f9d97d887505937f7ee3cc9f1e3ff"
Oct 29 23:03:49.676873 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:49.676850    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be840790c395da14e994e5eed7278ff4ab0f9d97d887505937f7ee3cc9f1e3ff\": container with ID starting with be840790c395da14e994e5eed7278ff4ab0f9d97d887505937f7ee3cc9f1e3ff not found: ID does not exist" containerID="be840790c395da14e994e5eed7278ff4ab0f9d97d887505937f7ee3cc9f1e3ff"
Oct 29 23:03:49.676924 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.676881    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be840790c395da14e994e5eed7278ff4ab0f9d97d887505937f7ee3cc9f1e3ff"} err="failed to get container status \"be840790c395da14e994e5eed7278ff4ab0f9d97d887505937f7ee3cc9f1e3ff\": rpc error: code = NotFound desc = could not find container \"be840790c395da14e994e5eed7278ff4ab0f9d97d887505937f7ee3cc9f1e3ff\": container with ID starting with be840790c395da14e994e5eed7278ff4ab0f9d97d887505937f7ee3cc9f1e3ff not found: ID does not exist"
Oct 29 23:03:49.741711 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.741665    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgw2f\" (UniqueName: \"kubernetes.io/projected/6f1b07f5-c3bd-4382-9318-2234d4e9093e-kube-api-access-dgw2f\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:49.741711 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.741706    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:49.741711 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.741715    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:49.741711 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.741723    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f1b07f5-c3bd-4382-9318-2234d4e9093e-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:49.985474 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.985432    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"]
Oct 29 23:03:49.997887 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:49.997856    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-80248b-openshift-default-5677b677f5-lx6cs"]
Oct 29 23:03:50.600857 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:03:50.600823    4559 logging.go:55] [core] [Channel #10437 SubChannel #10439]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.42:50052", ServerName: "10.134.0.42:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.42:50052: connect: connection refused"
Oct 29 23:03:50.810960 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:50.810878    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"
Oct 29 23:03:50.855110 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:50.855076    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b2c16e-ff49-4b9f-9e75-b0d2c4544532-kube-api-access-zkdch\" (UniqueName: \"kubernetes.io/projected/c7b2c16e-ff49-4b9f-9e75-b0d2c4544532-kube-api-access-zkdch\") pod \"c7b2c16e-ff49-4b9f-9e75-b0d2c4544532\" (UID: \"c7b2c16e-ff49-4b9f-9e75-b0d2c4544532\") "
Oct 29 23:03:50.855587 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:50.855417    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b2c16e-ff49-4b9f-9e75-b0d2c4544532-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b2c16e-ff49-4b9f-9e75-b0d2c4544532-plugins-config-volume\") pod \"c7b2c16e-ff49-4b9f-9e75-b0d2c4544532\" (UID: \"c7b2c16e-ff49-4b9f-9e75-b0d2c4544532\") "
Oct 29 23:03:50.856113 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:50.856087    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b2c16e-ff49-4b9f-9e75-b0d2c4544532-plugins-config-volume" pod "c7b2c16e-ff49-4b9f-9e75-b0d2c4544532" (UID: "c7b2c16e-ff49-4b9f-9e75-b0d2c4544532"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:50.858065 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:50.858035    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b2c16e-ff49-4b9f-9e75-b0d2c4544532-kube-api-access-zkdch" pod "c7b2c16e-ff49-4b9f-9e75-b0d2c4544532" (UID: "c7b2c16e-ff49-4b9f-9e75-b0d2c4544532"). InnerVolumeSpecName "kube-api-access-zkdch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:50.957875 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:50.957840    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkdch\" (UniqueName: \"kubernetes.io/projected/c7b2c16e-ff49-4b9f-9e75-b0d2c4544532-kube-api-access-zkdch\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:50.957875 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:50.957871    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b2c16e-ff49-4b9f-9e75-b0d2c4544532-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:51.229086 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:51.229046    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f1b07f5-c3bd-4382-9318-2234d4e9093e" path="/var/lib/kubelet/pods/6f1b07f5-c3bd-4382-9318-2234d4e9093e/volumes"
Oct 29 23:03:51.600519 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:51.600436    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx" podUID="c7b2c16e-ff49-4b9f-9e75-b0d2c4544532" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.42:50052\" within 1s: context deadline exceeded"
Oct 29 23:03:51.600662 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:51.600521    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"
Oct 29 23:03:51.655546 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:51.655497    4559 generic.go:356] "Generic (PLEG): container finished" podID="c7b2c16e-ff49-4b9f-9e75-b0d2c4544532" containerID="c2679dde734fc60ca409d39bda5f9fa652f7453dd299a0e52103a34ee7995826" exitCode=0
Oct 29 23:03:51.656051 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:51.655584    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"
Oct 29 23:03:51.656051 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:51.655591    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx" event={"ID":"c7b2c16e-ff49-4b9f-9e75-b0d2c4544532","Type":"ContainerDied","Data":"c2679dde734fc60ca409d39bda5f9fa652f7453dd299a0e52103a34ee7995826"}
Oct 29 23:03:51.656051 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:51.655655    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx" event={"ID":"c7b2c16e-ff49-4b9f-9e75-b0d2c4544532","Type":"ContainerDied","Data":"7e455be50cd03620324893fba0d8291202763e6ac121c109957dc403305a28b4"}
Oct 29 23:03:51.656051 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:51.655682    4559 scope.go:122] "RemoveContainer" containerID="c2679dde734fc60ca409d39bda5f9fa652f7453dd299a0e52103a34ee7995826"
Oct 29 23:03:51.668185 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:51.667984    4559 scope.go:122] "RemoveContainer" containerID="c2679dde734fc60ca409d39bda5f9fa652f7453dd299a0e52103a34ee7995826"
Oct 29 23:03:51.668351 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:51.668329    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2679dde734fc60ca409d39bda5f9fa652f7453dd299a0e52103a34ee7995826\": container with ID starting with c2679dde734fc60ca409d39bda5f9fa652f7453dd299a0e52103a34ee7995826 not found: ID does not exist" containerID="c2679dde734fc60ca409d39bda5f9fa652f7453dd299a0e52103a34ee7995826"
Oct 29 23:03:51.668424 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:51.668366    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2679dde734fc60ca409d39bda5f9fa652f7453dd299a0e52103a34ee7995826"} err="failed to get container status \"c2679dde734fc60ca409d39bda5f9fa652f7453dd299a0e52103a34ee7995826\": rpc error: code = NotFound desc = could not find container \"c2679dde734fc60ca409d39bda5f9fa652f7453dd299a0e52103a34ee7995826\": container with ID starting with c2679dde734fc60ca409d39bda5f9fa652f7453dd299a0e52103a34ee7995826 not found: ID does not exist"
Oct 29 23:03:51.707505 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:51.707464    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"]
Oct 29 23:03:51.720120 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:51.720080    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2ba10a-6d55f55df9-gfdpx"]
Oct 29 23:03:52.121919 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.121875    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"]
Oct 29 23:03:52.122195 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.122160    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v" podUID="fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" containerName="istio-proxy" containerID="cri-o://25807341e0d4bbbb160f73b582b3db4f1217baca011e958295ecf5d26d6d1cf0" gracePeriod=30
Oct 29 23:03:52.803568 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.803534    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"]
Oct 29 23:03:52.804093 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.804078    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6f1b07f5-c3bd-4382-9318-2234d4e9093e" containerName="istio-proxy"
Oct 29 23:03:52.804093 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.804093    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f1b07f5-c3bd-4382-9318-2234d4e9093e" containerName="istio-proxy"
Oct 29 23:03:52.804193 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.804117    4559 cpu_manager.go:414] "containerMap: removing container" podUID="63c90510-e767-4379-95c6-d1ea52e7180f" containerName="revoke-keys"
Oct 29 23:03:52.804193 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.804122    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c90510-e767-4379-95c6-d1ea52e7180f" containerName="revoke-keys"
Oct 29 23:03:52.804193 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.804146    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c7b2c16e-ff49-4b9f-9e75-b0d2c4544532" containerName="payload-pre-processing"
Oct 29 23:03:52.804193 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.804151    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b2c16e-ff49-4b9f-9e75-b0d2c4544532" containerName="payload-pre-processing"
Oct 29 23:03:52.804332 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.804218    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f1b07f5-c3bd-4382-9318-2234d4e9093e" containerName="istio-proxy"
Oct 29 23:03:52.804332 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.804227    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b2c16e-ff49-4b9f-9e75-b0d2c4544532" containerName="payload-pre-processing"
Oct 29 23:03:52.804332 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.804238    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c90510-e767-4379-95c6-d1ea52e7180f" containerName="revoke-keys"
Oct 29 23:03:52.809494 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.809470    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"
Oct 29 23:03:52.824541 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.824494    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:03:52.824974 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.824945    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:03:52.828880 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.828848    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"]
Oct 29 23:03:52.882650 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.882605    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2chtk\" (UniqueName: \"kubernetes.io/projected/a73d88c2-4afe-42d7-824b-c513cd56a432-kube-api-access-2chtk\") pod \"maas-api-revoke-keys-e2e-worker-w5-22d634-79klt\" (UID: \"a73d88c2-4afe-42d7-824b-c513cd56a432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"
Oct 29 23:03:52.882924 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.882676    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73d88c2-4afe-42d7-824b-c513cd56a432-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-22d634-79klt\" (UID: \"a73d88c2-4afe-42d7-824b-c513cd56a432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"
Oct 29 23:03:52.882982 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.882952    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73d88c2-4afe-42d7-824b-c513cd56a432-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-22d634-79klt\" (UID: \"a73d88c2-4afe-42d7-824b-c513cd56a432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"
Oct 29 23:03:52.984413 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.984368    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73d88c2-4afe-42d7-824b-c513cd56a432-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-22d634-79klt\" (UID: \"a73d88c2-4afe-42d7-824b-c513cd56a432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"
Oct 29 23:03:52.984413 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.984420    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2chtk\" (UniqueName: \"kubernetes.io/projected/a73d88c2-4afe-42d7-824b-c513cd56a432-kube-api-access-2chtk\") pod \"maas-api-revoke-keys-e2e-worker-w5-22d634-79klt\" (UID: \"a73d88c2-4afe-42d7-824b-c513cd56a432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"
Oct 29 23:03:52.984659 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.984463    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73d88c2-4afe-42d7-824b-c513cd56a432-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-22d634-79klt\" (UID: \"a73d88c2-4afe-42d7-824b-c513cd56a432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"
Oct 29 23:03:52.985256 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.985193    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73d88c2-4afe-42d7-824b-c513cd56a432-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-22d634-79klt\" (UID: \"a73d88c2-4afe-42d7-824b-c513cd56a432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"
Oct 29 23:03:52.987009 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:52.986986    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73d88c2-4afe-42d7-824b-c513cd56a432-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-22d634-79klt\" (UID: \"a73d88c2-4afe-42d7-824b-c513cd56a432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"
Oct 29 23:03:53.004161 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:53.004120    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2chtk\" (UniqueName: \"kubernetes.io/projected/a73d88c2-4afe-42d7-824b-c513cd56a432-kube-api-access-2chtk\") pod \"maas-api-revoke-keys-e2e-worker-w5-22d634-79klt\" (UID: \"a73d88c2-4afe-42d7-824b-c513cd56a432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"
Oct 29 23:03:53.125151 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:53.125047    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"
Oct 29 23:03:53.230138 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:53.230094    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b2c16e-ff49-4b9f-9e75-b0d2c4544532" path="/var/lib/kubelet/pods/c7b2c16e-ff49-4b9f-9e75-b0d2c4544532/volumes"
Oct 29 23:03:53.428267 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:53.428236    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"]
Oct 29 23:03:53.429567 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:03:53.429543    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73d88c2_4afe_42d7_824b_c513cd56a432.slice/crio-cdb4466b7d83de65200cf69e007b56cf5602acd085a58dc5460e1330e22b5f04 WatchSource:0}: Error finding container cdb4466b7d83de65200cf69e007b56cf5602acd085a58dc5460e1330e22b5f04: Status 404 returned error can't find the container with id cdb4466b7d83de65200cf69e007b56cf5602acd085a58dc5460e1330e22b5f04
Oct 29 23:03:53.671902 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:53.671865    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt" event={"ID":"a73d88c2-4afe-42d7-824b-c513cd56a432","Type":"ContainerStarted","Data":"cdb4466b7d83de65200cf69e007b56cf5602acd085a58dc5460e1330e22b5f04"}
Oct 29 23:03:54.650597 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:54.650556    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"]
Oct 29 23:03:54.651049 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:54.650880    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl" podUID="d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" containerName="istio-proxy" containerID="cri-o://4b7700b08a80e8b4595d7849c3dc29024d3ab55e8569b35715e6a0d3891bfce8" gracePeriod=30
Oct 29 23:03:54.682525 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:54.682484    4559 generic.go:356] "Generic (PLEG): container finished" podID="a73d88c2-4afe-42d7-824b-c513cd56a432" containerID="eea2d0617c92e318c5e97485f18930b9c96631a2534e8e71ea3897fe405eab1b" exitCode=0
Oct 29 23:03:54.682708 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:54.682568    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt" event={"ID":"a73d88c2-4afe-42d7-824b-c513cd56a432","Type":"ContainerDied","Data":"eea2d0617c92e318c5e97485f18930b9c96631a2534e8e71ea3897fe405eab1b"}
Oct 29 23:03:55.846695 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:55.846665    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"
Oct 29 23:03:55.921146 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:55.921109    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a73d88c2-4afe-42d7-824b-c513cd56a432-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73d88c2-4afe-42d7-824b-c513cd56a432-maas-api-xks-ca\") pod \"a73d88c2-4afe-42d7-824b-c513cd56a432\" (UID: \"a73d88c2-4afe-42d7-824b-c513cd56a432\") "
Oct 29 23:03:55.921330 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:55.921155    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a73d88c2-4afe-42d7-824b-c513cd56a432-kube-api-access-2chtk\" (UniqueName: \"kubernetes.io/projected/a73d88c2-4afe-42d7-824b-c513cd56a432-kube-api-access-2chtk\") pod \"a73d88c2-4afe-42d7-824b-c513cd56a432\" (UID: \"a73d88c2-4afe-42d7-824b-c513cd56a432\") "
Oct 29 23:03:55.921330 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:55.921176    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a73d88c2-4afe-42d7-824b-c513cd56a432-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73d88c2-4afe-42d7-824b-c513cd56a432-maas-api-openshift-ca\") pod \"a73d88c2-4afe-42d7-824b-c513cd56a432\" (UID: \"a73d88c2-4afe-42d7-824b-c513cd56a432\") "
Oct 29 23:03:55.921696 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:55.921657    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73d88c2-4afe-42d7-824b-c513cd56a432-maas-api-openshift-ca" pod "a73d88c2-4afe-42d7-824b-c513cd56a432" (UID: "a73d88c2-4afe-42d7-824b-c513cd56a432"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:55.924088 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:55.924060    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a73d88c2-4afe-42d7-824b-c513cd56a432-maas-api-xks-ca" pod "a73d88c2-4afe-42d7-824b-c513cd56a432" (UID: "a73d88c2-4afe-42d7-824b-c513cd56a432"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:55.924233 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:55.924063    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73d88c2-4afe-42d7-824b-c513cd56a432-kube-api-access-2chtk" pod "a73d88c2-4afe-42d7-824b-c513cd56a432" (UID: "a73d88c2-4afe-42d7-824b-c513cd56a432"). InnerVolumeSpecName "kube-api-access-2chtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:56.022841 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:56.022734    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73d88c2-4afe-42d7-824b-c513cd56a432-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:56.022841 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:56.022764    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2chtk\" (UniqueName: \"kubernetes.io/projected/a73d88c2-4afe-42d7-824b-c513cd56a432-kube-api-access-2chtk\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:56.022841 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:56.022773    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73d88c2-4afe-42d7-824b-c513cd56a432-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:56.702901 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:56.702864    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"
Oct 29 23:03:56.702901 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:56.702888    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt" event={"ID":"a73d88c2-4afe-42d7-824b-c513cd56a432","Type":"ContainerDied","Data":"cdb4466b7d83de65200cf69e007b56cf5602acd085a58dc5460e1330e22b5f04"}
Oct 29 23:03:56.703140 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:56.702942    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdb4466b7d83de65200cf69e007b56cf5602acd085a58dc5460e1330e22b5f04"
Oct 29 23:03:56.931632 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:56.931456    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:03:56.932291 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:56.932250    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v" podUID="fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:03:57.332612 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.332572    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n" podUID="2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:57.332788 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.332757    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh" podUID="7728dea4-21ee-478e-8fcc-c4632e1b350a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:57.433465 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.433439    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 23:03:57.541406 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.541357    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-kube-api-access-kppnp\" (UniqueName: \"kubernetes.io/projected/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-kube-api-access-kppnp\") pod \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") "
Oct 29 23:03:57.541406 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.541414    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-workload-certs\") pod \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") "
Oct 29 23:03:57.541702 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.541444    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-workload-socket\") pod \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") "
Oct 29 23:03:57.541702 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.541509    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-envoy\") pod \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") "
Oct 29 23:03:57.541702 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.541547    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-data\") pod \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") "
Oct 29 23:03:57.541702 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.541609    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-podinfo\") pod \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") "
Oct 29 23:03:57.541702 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.541667    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-token\" (UniqueName: \"kubernetes.io/projected/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-token\") pod \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") "
Oct 29 23:03:57.542025 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.541693    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-workload-certs" pod "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" (UID: "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.542025 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.541739    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istiod-ca-cert\") pod \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") "
Oct 29 23:03:57.542025 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.541794    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-credential-socket\") pod \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\" (UID: \"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6\") "
Oct 29 23:03:57.542025 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.541979    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-data" pod "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" (UID: "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.542284 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.542203    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-workload-socket" pod "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" (UID: "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.542345 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.542306    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istiod-ca-cert" pod "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" (UID: "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:57.542456 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.542430    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-credential-socket" pod "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" (UID: "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.542612 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.542487    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.542722 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.542710    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.542932 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.542912    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.542932 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.542933    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.544849 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.544773    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-podinfo" pod "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" (UID: "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:57.544849 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.544776    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-token" pod "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" (UID: "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:57.544849 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.544795    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-kube-api-access-kppnp" pod "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" (UID: "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6"). InnerVolumeSpecName "kube-api-access-kppnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:57.545098 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.545080    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-envoy" pod "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" (UID: "fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.644705 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.644649    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.644705 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.644699    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kppnp\" (UniqueName: \"kubernetes.io/projected/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-kube-api-access-kppnp\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.644705 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.644715    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.645088 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.644729    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.645088 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.644742    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.712316 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.712226    4559 generic.go:356] "Generic (PLEG): container finished" podID="fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" containerID="25807341e0d4bbbb160f73b582b3db4f1217baca011e958295ecf5d26d6d1cf0" exitCode=0
Oct 29 23:03:57.712316 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.712283    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v" event={"ID":"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6","Type":"ContainerDied","Data":"25807341e0d4bbbb160f73b582b3db4f1217baca011e958295ecf5d26d6d1cf0"}
Oct 29 23:03:57.712525 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.712322    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v" event={"ID":"fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6","Type":"ContainerDied","Data":"39f35ed5b5f7f7d85abcb0dd9143c07d0d2a978f32f1869fdc71b8824c58cb43"}
Oct 29 23:03:57.712525 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.712342    4559 scope.go:122] "RemoveContainer" containerID="25807341e0d4bbbb160f73b582b3db4f1217baca011e958295ecf5d26d6d1cf0"
Oct 29 23:03:57.712525 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.712285    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"
Oct 29 23:03:57.723458 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.723437    4559 scope.go:122] "RemoveContainer" containerID="234e547bc5821a880a916bd6b684de6dbee12b5377dffafb082c7b20aaf3ea48"
Oct 29 23:03:57.733586 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.733557    4559 scope.go:122] "RemoveContainer" containerID="25807341e0d4bbbb160f73b582b3db4f1217baca011e958295ecf5d26d6d1cf0"
Oct 29 23:03:57.734713 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:57.734681    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25807341e0d4bbbb160f73b582b3db4f1217baca011e958295ecf5d26d6d1cf0\": container with ID starting with 25807341e0d4bbbb160f73b582b3db4f1217baca011e958295ecf5d26d6d1cf0 not found: ID does not exist" containerID="25807341e0d4bbbb160f73b582b3db4f1217baca011e958295ecf5d26d6d1cf0"
Oct 29 23:03:57.734801 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.734729    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25807341e0d4bbbb160f73b582b3db4f1217baca011e958295ecf5d26d6d1cf0"} err="failed to get container status \"25807341e0d4bbbb160f73b582b3db4f1217baca011e958295ecf5d26d6d1cf0\": rpc error: code = NotFound desc = could not find container \"25807341e0d4bbbb160f73b582b3db4f1217baca011e958295ecf5d26d6d1cf0\": container with ID starting with 25807341e0d4bbbb160f73b582b3db4f1217baca011e958295ecf5d26d6d1cf0 not found: ID does not exist"
Oct 29 23:03:57.734801 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.734756    4559 scope.go:122] "RemoveContainer" containerID="234e547bc5821a880a916bd6b684de6dbee12b5377dffafb082c7b20aaf3ea48"
Oct 29 23:03:57.735114 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:57.735086    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"234e547bc5821a880a916bd6b684de6dbee12b5377dffafb082c7b20aaf3ea48\": container with ID starting with 234e547bc5821a880a916bd6b684de6dbee12b5377dffafb082c7b20aaf3ea48 not found: ID does not exist" containerID="234e547bc5821a880a916bd6b684de6dbee12b5377dffafb082c7b20aaf3ea48"
Oct 29 23:03:57.735189 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.735119    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"234e547bc5821a880a916bd6b684de6dbee12b5377dffafb082c7b20aaf3ea48"} err="failed to get container status \"234e547bc5821a880a916bd6b684de6dbee12b5377dffafb082c7b20aaf3ea48\": rpc error: code = NotFound desc = could not find container \"234e547bc5821a880a916bd6b684de6dbee12b5377dffafb082c7b20aaf3ea48\": container with ID starting with 234e547bc5821a880a916bd6b684de6dbee12b5377dffafb082c7b20aaf3ea48 not found: ID does not exist"
Oct 29 23:03:57.766270 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.766230    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"]
Oct 29 23:03:57.778934 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:57.778900    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-82f2v"]
Oct 29 23:03:58.092126 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.092095    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"
Oct 29 23:03:58.129489 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.129462    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"
Oct 29 23:03:58.154938 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.154910    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7728dea4-21ee-478e-8fcc-c4632e1b350a-kube-api-access-x8vc9\" (UniqueName: \"kubernetes.io/projected/7728dea4-21ee-478e-8fcc-c4632e1b350a-kube-api-access-x8vc9\") pod \"7728dea4-21ee-478e-8fcc-c4632e1b350a\" (UID: \"7728dea4-21ee-478e-8fcc-c4632e1b350a\") "
Oct 29 23:03:58.155115 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.155047    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7728dea4-21ee-478e-8fcc-c4632e1b350a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7728dea4-21ee-478e-8fcc-c4632e1b350a-plugins-config-volume\") pod \"7728dea4-21ee-478e-8fcc-c4632e1b350a\" (UID: \"7728dea4-21ee-478e-8fcc-c4632e1b350a\") "
Oct 29 23:03:58.156134 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.156090    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7728dea4-21ee-478e-8fcc-c4632e1b350a-plugins-config-volume" pod "7728dea4-21ee-478e-8fcc-c4632e1b350a" (UID: "7728dea4-21ee-478e-8fcc-c4632e1b350a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:58.187422 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.187373    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7728dea4-21ee-478e-8fcc-c4632e1b350a-kube-api-access-x8vc9" pod "7728dea4-21ee-478e-8fcc-c4632e1b350a" (UID: "7728dea4-21ee-478e-8fcc-c4632e1b350a"). InnerVolumeSpecName "kube-api-access-x8vc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:58.256865 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.256764    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0-kube-api-access-6r6cx\" (UniqueName: \"kubernetes.io/projected/2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0-kube-api-access-6r6cx\") pod \"2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0\" (UID: \"2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0\") "
Oct 29 23:03:58.257047 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.256912    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0-plugins-config-volume\") pod \"2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0\" (UID: \"2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0\") "
Oct 29 23:03:58.257321 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.257302    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8vc9\" (UniqueName: \"kubernetes.io/projected/7728dea4-21ee-478e-8fcc-c4632e1b350a-kube-api-access-x8vc9\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.257397 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.257325    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7728dea4-21ee-478e-8fcc-c4632e1b350a-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.257397 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.257327    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0-plugins-config-volume" pod "2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0" (UID: "2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:58.259360 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.259331    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0-kube-api-access-6r6cx" pod "2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0" (UID: "2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0"). InnerVolumeSpecName "kube-api-access-6r6cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:58.359161 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.359111    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r6cx\" (UniqueName: \"kubernetes.io/projected/2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0-kube-api-access-6r6cx\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.359161 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.359159    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.364304 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.364263    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:03:58.364477 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.364330    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl" podUID="d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:03:58.717184 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.717147    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:03:58.721477 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.721446    4559 generic.go:356] "Generic (PLEG): container finished" podID="2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0" containerID="8ffc843b4979e16d3734b097d84e9798f3d086242c47d4024e5b390806845ff3" exitCode=0
Oct 29 23:03:58.721653 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.721489    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"
Oct 29 23:03:58.721653 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.721530    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n" event={"ID":"2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0","Type":"ContainerDied","Data":"8ffc843b4979e16d3734b097d84e9798f3d086242c47d4024e5b390806845ff3"}
Oct 29 23:03:58.721653 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.721586    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n" event={"ID":"2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0","Type":"ContainerDied","Data":"b831ce0e1df06cc6ca900702b1d36521a225162f4842e2db0fea959e4da51e2d"}
Oct 29 23:03:58.721653 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.721617    4559 scope.go:122] "RemoveContainer" containerID="8ffc843b4979e16d3734b097d84e9798f3d086242c47d4024e5b390806845ff3"
Oct 29 23:03:58.724532 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.724506    4559 generic.go:356] "Generic (PLEG): container finished" podID="7728dea4-21ee-478e-8fcc-c4632e1b350a" containerID="25e990b39d274eaf0eedb28be6d181b7ba8043740fa224f2555de6b4c93207e8" exitCode=0
Oct 29 23:03:58.724657 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.724533    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh" event={"ID":"7728dea4-21ee-478e-8fcc-c4632e1b350a","Type":"ContainerDied","Data":"25e990b39d274eaf0eedb28be6d181b7ba8043740fa224f2555de6b4c93207e8"}
Oct 29 23:03:58.724657 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.724548    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"
Oct 29 23:03:58.724657 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.724581    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh" event={"ID":"7728dea4-21ee-478e-8fcc-c4632e1b350a","Type":"ContainerDied","Data":"6be1ddbef22c67c18ff959397bed357a682285da004c6cf73806b2740b3620f7"}
Oct 29 23:03:58.733764 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.733743    4559 scope.go:122] "RemoveContainer" containerID="8ffc843b4979e16d3734b097d84e9798f3d086242c47d4024e5b390806845ff3"
Oct 29 23:03:58.734205 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:58.734140    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ffc843b4979e16d3734b097d84e9798f3d086242c47d4024e5b390806845ff3\": container with ID starting with 8ffc843b4979e16d3734b097d84e9798f3d086242c47d4024e5b390806845ff3 not found: ID does not exist" containerID="8ffc843b4979e16d3734b097d84e9798f3d086242c47d4024e5b390806845ff3"
Oct 29 23:03:58.734205 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.734180    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ffc843b4979e16d3734b097d84e9798f3d086242c47d4024e5b390806845ff3"} err="failed to get container status \"8ffc843b4979e16d3734b097d84e9798f3d086242c47d4024e5b390806845ff3\": rpc error: code = NotFound desc = could not find container \"8ffc843b4979e16d3734b097d84e9798f3d086242c47d4024e5b390806845ff3\": container with ID starting with 8ffc843b4979e16d3734b097d84e9798f3d086242c47d4024e5b390806845ff3 not found: ID does not exist"
Oct 29 23:03:58.734205 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.734206    4559 scope.go:122] "RemoveContainer" containerID="25e990b39d274eaf0eedb28be6d181b7ba8043740fa224f2555de6b4c93207e8"
Oct 29 23:03:58.744129 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.744107    4559 scope.go:122] "RemoveContainer" containerID="25e990b39d274eaf0eedb28be6d181b7ba8043740fa224f2555de6b4c93207e8"
Oct 29 23:03:58.744451 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:03:58.744428    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e990b39d274eaf0eedb28be6d181b7ba8043740fa224f2555de6b4c93207e8\": container with ID starting with 25e990b39d274eaf0eedb28be6d181b7ba8043740fa224f2555de6b4c93207e8 not found: ID does not exist" containerID="25e990b39d274eaf0eedb28be6d181b7ba8043740fa224f2555de6b4c93207e8"
Oct 29 23:03:58.744512 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.744463    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e990b39d274eaf0eedb28be6d181b7ba8043740fa224f2555de6b4c93207e8"} err="failed to get container status \"25e990b39d274eaf0eedb28be6d181b7ba8043740fa224f2555de6b4c93207e8\": rpc error: code = NotFound desc = could not find container \"25e990b39d274eaf0eedb28be6d181b7ba8043740fa224f2555de6b4c93207e8\": container with ID starting with 25e990b39d274eaf0eedb28be6d181b7ba8043740fa224f2555de6b4c93207e8 not found: ID does not exist"
Oct 29 23:03:58.773197 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.773163    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"]
Oct 29 23:03:58.775530 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.775498    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-6fc6bd-cf68d45f8-22lvh"]
Oct 29 23:03:58.807678 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.807644    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"]
Oct 29 23:03:58.810074 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:58.810050    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-6fc6bd-78c4d8b54c-qg69n"]
Oct 29 23:03:59.230724 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:59.230678    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0" path="/var/lib/kubelet/pods/2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0/volumes"
Oct 29 23:03:59.231387 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:59.231361    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7728dea4-21ee-478e-8fcc-c4632e1b350a" path="/var/lib/kubelet/pods/7728dea4-21ee-478e-8fcc-c4632e1b350a/volumes"
Oct 29 23:03:59.232092 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:59.232065    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" path="/var/lib/kubelet/pods/fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6/volumes"
Oct 29 23:03:59.923927 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:03:59.923856    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 23:04:00.075853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.075801    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-credential-socket\") pod \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") "
Oct 29 23:04:00.075853 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.075858    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-workload-socket\") pod \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") "
Oct 29 23:04:00.076114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.075892    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-data\") pod \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") "
Oct 29 23:04:00.076114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.075913    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-workload-certs\") pod \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") "
Oct 29 23:04:00.076114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.075963    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istiod-ca-cert\") pod \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") "
Oct 29 23:04:00.076114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.076015    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-token\" (UniqueName: \"kubernetes.io/projected/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-token\") pod \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") "
Oct 29 23:04:00.076114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.076045    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-envoy\") pod \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") "
Oct 29 23:04:00.076395 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.076123    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-kube-api-access-ztjkh\" (UniqueName: \"kubernetes.io/projected/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-kube-api-access-ztjkh\") pod \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") "
Oct 29 23:04:00.076395 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.076137    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-credential-socket" pod "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" (UID: "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:00.076395 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.076154    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-workload-socket" pod "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" (UID: "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:00.076395 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.076224    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-podinfo\") pod \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\" (UID: \"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d\") "
Oct 29 23:04:00.076395 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.076296    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-data" pod "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" (UID: "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:00.076615 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.076481    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istiod-ca-cert" pod "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" (UID: "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:00.076701 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.076678    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-workload-certs" pod "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" (UID: "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:00.076788 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.076768    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.076864 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.076797    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.076864 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.076830    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.076864 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.076842    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.078527 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.078493    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-podinfo" pod "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" (UID: "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:00.078527 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.078507    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-envoy" pod "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" (UID: "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:00.078987 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.078959    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-token" pod "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" (UID: "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:00.078987 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.078972    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-kube-api-access-ztjkh" pod "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" (UID: "d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d"). InnerVolumeSpecName "kube-api-access-ztjkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:00.177874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.177744    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.177874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.177781    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.177874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.177790    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.177874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.177798    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.177874 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.177843    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztjkh\" (UniqueName: \"kubernetes.io/projected/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d-kube-api-access-ztjkh\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:00.742966 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.742933    4559 generic.go:356] "Generic (PLEG): container finished" podID="d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" containerID="4b7700b08a80e8b4595d7849c3dc29024d3ab55e8569b35715e6a0d3891bfce8" exitCode=0
Oct 29 23:04:00.743405 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.742986    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"
Oct 29 23:04:00.743405 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.742994    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl" event={"ID":"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d","Type":"ContainerDied","Data":"4b7700b08a80e8b4595d7849c3dc29024d3ab55e8569b35715e6a0d3891bfce8"}
Oct 29 23:04:00.743405 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.743037    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl" event={"ID":"d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d","Type":"ContainerDied","Data":"202b01fa078656c6e4d90b0c3e962479af81969ae7850af77ccf0fdc433ec283"}
Oct 29 23:04:00.743405 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.743059    4559 scope.go:122] "RemoveContainer" containerID="4b7700b08a80e8b4595d7849c3dc29024d3ab55e8569b35715e6a0d3891bfce8"
Oct 29 23:04:00.754019 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.753997    4559 scope.go:122] "RemoveContainer" containerID="4b7700b08a80e8b4595d7849c3dc29024d3ab55e8569b35715e6a0d3891bfce8"
Oct 29 23:04:00.754323 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:04:00.754301    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b7700b08a80e8b4595d7849c3dc29024d3ab55e8569b35715e6a0d3891bfce8\": container with ID starting with 4b7700b08a80e8b4595d7849c3dc29024d3ab55e8569b35715e6a0d3891bfce8 not found: ID does not exist" containerID="4b7700b08a80e8b4595d7849c3dc29024d3ab55e8569b35715e6a0d3891bfce8"
Oct 29 23:04:00.754401 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.754331    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b7700b08a80e8b4595d7849c3dc29024d3ab55e8569b35715e6a0d3891bfce8"} err="failed to get container status \"4b7700b08a80e8b4595d7849c3dc29024d3ab55e8569b35715e6a0d3891bfce8\": rpc error: code = NotFound desc = could not find container \"4b7700b08a80e8b4595d7849c3dc29024d3ab55e8569b35715e6a0d3891bfce8\": container with ID starting with 4b7700b08a80e8b4595d7849c3dc29024d3ab55e8569b35715e6a0d3891bfce8 not found: ID does not exist"
Oct 29 23:04:00.808971 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.808928    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"]
Oct 29 23:04:00.820932 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:00.820892    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-787nl"]
Oct 29 23:04:01.227679 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:01.227643    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" path="/var/lib/kubelet/pods/d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d/volumes"
Oct 29 23:04:04.413969 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.413927    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"]
Oct 29 23:04:04.414483 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.414140    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5" podUID="c1884171-dc00-4a39-8181-75468d0ba179" containerName="maas-api" containerID="cri-o://fa2a38eaff5b40cb4a2b9fc17ac80bfa9990079e809a1290904c55e3b1d07414" gracePeriod=30
Oct 29 23:04:04.562579 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.562542    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"]
Oct 29 23:04:04.562783 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.562760    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q" podUID="303751f6-3328-401c-9f2c-4970a54ae09c" containerName="payload-processing" containerID="cri-o://33e8a0701e5057ce81e167e2ec82974736ba087eb315e27b7d5042443f21b7eb" gracePeriod=30
Oct 29 23:04:04.583110 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.583078    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"]
Oct 29 23:04:04.583352 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.583323    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw" podUID="e367910b-b67d-499e-a1dd-3eda32d1fa33" containerName="payload-pre-processing" containerID="cri-o://d7c36dad33f834ab05258be2df62782b7e914f3c095fdb128b1e48ec9179d7db" gracePeriod=30
Oct 29 23:04:04.723159 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.723128    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:04:04.781128 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.779997    4559 generic.go:356] "Generic (PLEG): container finished" podID="c1884171-dc00-4a39-8181-75468d0ba179" containerID="fa2a38eaff5b40cb4a2b9fc17ac80bfa9990079e809a1290904c55e3b1d07414" exitCode=0
Oct 29 23:04:04.781128 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.780060    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"
Oct 29 23:04:04.781128 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.780046    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5" event={"ID":"c1884171-dc00-4a39-8181-75468d0ba179","Type":"ContainerDied","Data":"fa2a38eaff5b40cb4a2b9fc17ac80bfa9990079e809a1290904c55e3b1d07414"}
Oct 29 23:04:04.781128 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.780194    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5" event={"ID":"c1884171-dc00-4a39-8181-75468d0ba179","Type":"ContainerDied","Data":"bda9161b697ca5f74bd51b546dc24bf8599212d5abc0bac2fbca4bd60f5a46a0"}
Oct 29 23:04:04.781128 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.780227    4559 scope.go:122] "RemoveContainer" containerID="fa2a38eaff5b40cb4a2b9fc17ac80bfa9990079e809a1290904c55e3b1d07414"
Oct 29 23:04:04.783015 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.782856    4559 generic.go:356] "Generic (PLEG): container finished" podID="e367910b-b67d-499e-a1dd-3eda32d1fa33" containerID="d7c36dad33f834ab05258be2df62782b7e914f3c095fdb128b1e48ec9179d7db" exitCode=0
Oct 29 23:04:04.783015 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.782961    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw" event={"ID":"e367910b-b67d-499e-a1dd-3eda32d1fa33","Type":"ContainerDied","Data":"d7c36dad33f834ab05258be2df62782b7e914f3c095fdb128b1e48ec9179d7db"}
Oct 29 23:04:04.785731 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.785058    4559 generic.go:356] "Generic (PLEG): container finished" podID="303751f6-3328-401c-9f2c-4970a54ae09c" containerID="33e8a0701e5057ce81e167e2ec82974736ba087eb315e27b7d5042443f21b7eb" exitCode=0
Oct 29 23:04:04.785731 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.785163    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q" event={"ID":"303751f6-3328-401c-9f2c-4970a54ae09c","Type":"ContainerDied","Data":"33e8a0701e5057ce81e167e2ec82974736ba087eb315e27b7d5042443f21b7eb"}
Oct 29 23:04:04.792846 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.792784    4559 scope.go:122] "RemoveContainer" containerID="fa2a38eaff5b40cb4a2b9fc17ac80bfa9990079e809a1290904c55e3b1d07414"
Oct 29 23:04:04.793202 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:04:04.793176    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2a38eaff5b40cb4a2b9fc17ac80bfa9990079e809a1290904c55e3b1d07414\": container with ID starting with fa2a38eaff5b40cb4a2b9fc17ac80bfa9990079e809a1290904c55e3b1d07414 not found: ID does not exist" containerID="fa2a38eaff5b40cb4a2b9fc17ac80bfa9990079e809a1290904c55e3b1d07414"
Oct 29 23:04:04.793281 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.793209    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2a38eaff5b40cb4a2b9fc17ac80bfa9990079e809a1290904c55e3b1d07414"} err="failed to get container status \"fa2a38eaff5b40cb4a2b9fc17ac80bfa9990079e809a1290904c55e3b1d07414\": rpc error: code = NotFound desc = could not find container \"fa2a38eaff5b40cb4a2b9fc17ac80bfa9990079e809a1290904c55e3b1d07414\": container with ID starting with fa2a38eaff5b40cb4a2b9fc17ac80bfa9990079e809a1290904c55e3b1d07414 not found: ID does not exist"
Oct 29 23:04:04.835112 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.835068    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"
Oct 29 23:04:04.841897 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.841862    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1884171-dc00-4a39-8181-75468d0ba179-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1884171-dc00-4a39-8181-75468d0ba179-openshift-service-ca\") pod \"c1884171-dc00-4a39-8181-75468d0ba179\" (UID: \"c1884171-dc00-4a39-8181-75468d0ba179\") "
Oct 29 23:04:04.842070 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.841917    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1884171-dc00-4a39-8181-75468d0ba179-kube-api-access-k88mr\" (UniqueName: \"kubernetes.io/projected/c1884171-dc00-4a39-8181-75468d0ba179-kube-api-access-k88mr\") pod \"c1884171-dc00-4a39-8181-75468d0ba179\" (UID: \"c1884171-dc00-4a39-8181-75468d0ba179\") "
Oct 29 23:04:04.842070 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.841977    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1884171-dc00-4a39-8181-75468d0ba179-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1884171-dc00-4a39-8181-75468d0ba179-maas-api-tls\") pod \"c1884171-dc00-4a39-8181-75468d0ba179\" (UID: \"c1884171-dc00-4a39-8181-75468d0ba179\") "
Oct 29 23:04:04.842321 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.842113    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1884171-dc00-4a39-8181-75468d0ba179-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1884171-dc00-4a39-8181-75468d0ba179-metrics-certs\") pod \"c1884171-dc00-4a39-8181-75468d0ba179\" (UID: \"c1884171-dc00-4a39-8181-75468d0ba179\") "
Oct 29 23:04:04.842382 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.842333    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1884171-dc00-4a39-8181-75468d0ba179-openshift-service-ca" pod "c1884171-dc00-4a39-8181-75468d0ba179" (UID: "c1884171-dc00-4a39-8181-75468d0ba179"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:04.843324 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.843301    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1884171-dc00-4a39-8181-75468d0ba179-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:04.845654 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.845196    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1884171-dc00-4a39-8181-75468d0ba179-maas-api-tls" pod "c1884171-dc00-4a39-8181-75468d0ba179" (UID: "c1884171-dc00-4a39-8181-75468d0ba179"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:04.845654 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.845353    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1884171-dc00-4a39-8181-75468d0ba179-metrics-certs" pod "c1884171-dc00-4a39-8181-75468d0ba179" (UID: "c1884171-dc00-4a39-8181-75468d0ba179"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:04.849968 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.849937    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1884171-dc00-4a39-8181-75468d0ba179-kube-api-access-k88mr" pod "c1884171-dc00-4a39-8181-75468d0ba179" (UID: "c1884171-dc00-4a39-8181-75468d0ba179"). InnerVolumeSpecName "kube-api-access-k88mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:04.941063 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.941028    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"
Oct 29 23:04:04.944560 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.944482    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/303751f6-3328-401c-9f2c-4970a54ae09c-kube-api-access-9dmmf\" (UniqueName: \"kubernetes.io/projected/303751f6-3328-401c-9f2c-4970a54ae09c-kube-api-access-9dmmf\") pod \"303751f6-3328-401c-9f2c-4970a54ae09c\" (UID: \"303751f6-3328-401c-9f2c-4970a54ae09c\") "
Oct 29 23:04:04.944560 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.944552    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/303751f6-3328-401c-9f2c-4970a54ae09c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303751f6-3328-401c-9f2c-4970a54ae09c-plugins-config-volume\") pod \"303751f6-3328-401c-9f2c-4970a54ae09c\" (UID: \"303751f6-3328-401c-9f2c-4970a54ae09c\") "
Oct 29 23:04:04.944952 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.944926    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/303751f6-3328-401c-9f2c-4970a54ae09c-plugins-config-volume" pod "303751f6-3328-401c-9f2c-4970a54ae09c" (UID: "303751f6-3328-401c-9f2c-4970a54ae09c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:04.945040 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.944962    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1884171-dc00-4a39-8181-75468d0ba179-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:04.945040 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.944979    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1884171-dc00-4a39-8181-75468d0ba179-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:04.945040 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.944994    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k88mr\" (UniqueName: \"kubernetes.io/projected/c1884171-dc00-4a39-8181-75468d0ba179-kube-api-access-k88mr\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:04.946866 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:04.946801    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303751f6-3328-401c-9f2c-4970a54ae09c-kube-api-access-9dmmf" pod "303751f6-3328-401c-9f2c-4970a54ae09c" (UID: "303751f6-3328-401c-9f2c-4970a54ae09c"). InnerVolumeSpecName "kube-api-access-9dmmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:05.046787 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.046752    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e367910b-b67d-499e-a1dd-3eda32d1fa33-kube-api-access-5b5mf\" (UniqueName: \"kubernetes.io/projected/e367910b-b67d-499e-a1dd-3eda32d1fa33-kube-api-access-5b5mf\") pod \"e367910b-b67d-499e-a1dd-3eda32d1fa33\" (UID: \"e367910b-b67d-499e-a1dd-3eda32d1fa33\") "
Oct 29 23:04:05.047029 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.046873    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e367910b-b67d-499e-a1dd-3eda32d1fa33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e367910b-b67d-499e-a1dd-3eda32d1fa33-plugins-config-volume\") pod \"e367910b-b67d-499e-a1dd-3eda32d1fa33\" (UID: \"e367910b-b67d-499e-a1dd-3eda32d1fa33\") "
Oct 29 23:04:05.047101 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.047088    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dmmf\" (UniqueName: \"kubernetes.io/projected/303751f6-3328-401c-9f2c-4970a54ae09c-kube-api-access-9dmmf\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:05.047156 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.047101    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303751f6-3328-401c-9f2c-4970a54ae09c-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:05.047313 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.047288    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e367910b-b67d-499e-a1dd-3eda32d1fa33-plugins-config-volume" pod "e367910b-b67d-499e-a1dd-3eda32d1fa33" (UID: "e367910b-b67d-499e-a1dd-3eda32d1fa33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:05.049211 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.049183    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e367910b-b67d-499e-a1dd-3eda32d1fa33-kube-api-access-5b5mf" pod "e367910b-b67d-499e-a1dd-3eda32d1fa33" (UID: "e367910b-b67d-499e-a1dd-3eda32d1fa33"). InnerVolumeSpecName "kube-api-access-5b5mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:05.125378 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.125337    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"]
Oct 29 23:04:05.128598 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.128554    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"]
Oct 29 23:04:05.131104 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.131076    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d6d4e-rtfjk"]
Oct 29 23:04:05.135540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.135511    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6d4e-6946c66cf4-pvzj5"]
Oct 29 23:04:05.148520 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.148483    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e367910b-b67d-499e-a1dd-3eda32d1fa33-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:05.148520 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.148518    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b5mf\" (UniqueName: \"kubernetes.io/projected/e367910b-b67d-499e-a1dd-3eda32d1fa33-kube-api-access-5b5mf\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:05.227014 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.226932    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c90510-e767-4379-95c6-d1ea52e7180f" path="/var/lib/kubelet/pods/63c90510-e767-4379-95c6-d1ea52e7180f/volumes"
Oct 29 23:04:05.227372 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.227358    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1884171-dc00-4a39-8181-75468d0ba179" path="/var/lib/kubelet/pods/c1884171-dc00-4a39-8181-75468d0ba179/volumes"
Oct 29 23:04:05.794394 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.794360    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"
Oct 29 23:04:05.794394 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.794372    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw" event={"ID":"e367910b-b67d-499e-a1dd-3eda32d1fa33","Type":"ContainerDied","Data":"33e70e7280adfeb943969ceffe263260d5c48cb6499a82c1cf35573771cac946"}
Oct 29 23:04:05.795003 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.794442    4559 scope.go:122] "RemoveContainer" containerID="d7c36dad33f834ab05258be2df62782b7e914f3c095fdb128b1e48ec9179d7db"
Oct 29 23:04:05.796013 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.795986    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q" event={"ID":"303751f6-3328-401c-9f2c-4970a54ae09c","Type":"ContainerDied","Data":"4e3bde41c9ef565959d5afb1de3212b6320944a3f16dd022fd1ef5fb3690b986"}
Oct 29 23:04:05.796136 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.796014    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"
Oct 29 23:04:05.804711 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.804692    4559 scope.go:122] "RemoveContainer" containerID="33e8a0701e5057ce81e167e2ec82974736ba087eb315e27b7d5042443f21b7eb"
Oct 29 23:04:05.845527 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.845495    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"]
Oct 29 23:04:05.858049 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.858018    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-767bdcc655-qhqnw"]
Oct 29 23:04:05.882716 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.882681    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"]
Oct 29 23:04:05.886194 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:05.886163    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-65f69f95b8-qb57q"]
Oct 29 23:04:07.227267 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:07.227229    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="303751f6-3328-401c-9f2c-4970a54ae09c" path="/var/lib/kubelet/pods/303751f6-3328-401c-9f2c-4970a54ae09c/volumes"
Oct 29 23:04:07.227758 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:07.227740    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e367910b-b67d-499e-a1dd-3eda32d1fa33" path="/var/lib/kubelet/pods/e367910b-b67d-499e-a1dd-3eda32d1fa33/volumes"
Oct 29 23:04:11.142664 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.142620    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"]
Oct 29 23:04:11.143358 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143334    4559 cpu_manager.go:414] "containerMap: removing container" podUID="303751f6-3328-401c-9f2c-4970a54ae09c" containerName="payload-processing"
Oct 29 23:04:11.143452 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143360    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="303751f6-3328-401c-9f2c-4970a54ae09c" containerName="payload-processing"
Oct 29 23:04:11.143452 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143377    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0" containerName="payload-pre-processing"
Oct 29 23:04:11.143452 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143384    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0" containerName="payload-pre-processing"
Oct 29 23:04:11.143452 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143410    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6f1b07f5-c3bd-4382-9318-2234d4e9093e" containerName="istio-proxy"
Oct 29 23:04:11.143452 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143418    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f1b07f5-c3bd-4382-9318-2234d4e9093e" containerName="istio-proxy"
Oct 29 23:04:11.143452 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143429    4559 cpu_manager.go:414] "containerMap: removing container" podUID="fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" containerName="istio-proxy"
Oct 29 23:04:11.143452 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143435    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" containerName="istio-proxy"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143462    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7728dea4-21ee-478e-8fcc-c4632e1b350a" containerName="payload-processing"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143470    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7728dea4-21ee-478e-8fcc-c4632e1b350a" containerName="payload-processing"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143486    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" containerName="istio-proxy"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143493    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" containerName="istio-proxy"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143509    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c1884171-dc00-4a39-8181-75468d0ba179" containerName="maas-api"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143524    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1884171-dc00-4a39-8181-75468d0ba179" containerName="maas-api"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143540    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a73d88c2-4afe-42d7-824b-c513cd56a432" containerName="revoke-keys"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143547    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a73d88c2-4afe-42d7-824b-c513cd56a432" containerName="revoke-keys"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143562    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e367910b-b67d-499e-a1dd-3eda32d1fa33" containerName="payload-pre-processing"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143570    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e367910b-b67d-499e-a1dd-3eda32d1fa33" containerName="payload-pre-processing"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143676    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c5c47b-9b0c-4d07-bfca-5eacb7d77f0d" containerName="istio-proxy"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143694    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1884171-dc00-4a39-8181-75468d0ba179" containerName="maas-api"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143706    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a73d88c2-4afe-42d7-824b-c513cd56a432" containerName="revoke-keys"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143719    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e367910b-b67d-499e-a1dd-3eda32d1fa33" containerName="payload-pre-processing"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143729    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f1b07f5-c3bd-4382-9318-2234d4e9093e" containerName="istio-proxy"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143737    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="303751f6-3328-401c-9f2c-4970a54ae09c" containerName="payload-processing"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143747    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4bcb98-02b3-4b0d-9f4f-6b5e6976fdc0" containerName="payload-pre-processing"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143759    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" containerName="istio-proxy"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143772    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7728dea4-21ee-478e-8fcc-c4632e1b350a" containerName="payload-processing"
Oct 29 23:04:11.143851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.143780    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" containerName="istio-proxy"
Oct 29 23:04:11.149070 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.149044    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"
Oct 29 23:04:11.163276 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.163244    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"]
Oct 29 23:04:11.163925 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.163902    4559 cpu_manager.go:414] "containerMap: removing container" podUID="fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" containerName="istio-proxy"
Oct 29 23:04:11.163925 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.163924    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3385fd-f8ce-46b7-8ea2-cffddd2cd9a6" containerName="istio-proxy"
Oct 29 23:04:11.169168 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.169144    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"
Oct 29 23:04:11.173042 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.173009    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:04:11.173042 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.173020    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:04:11.173264 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.173206    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:04:11.176149 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.176124    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:04:11.179180 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.179157    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"]
Oct 29 23:04:11.189538 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.189508    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"]
Oct 29 23:04:11.219867 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.219802    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b244d1-56ea-4f24-a7e0-527308b9aa4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efaeed0e-bm78q\" (UID: \"05b244d1-56ea-4f24-a7e0-527308b9aa4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"
Oct 29 23:04:11.220037 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.219871    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvnsp\" (UniqueName: \"kubernetes.io/projected/05b244d1-56ea-4f24-a7e0-527308b9aa4a-kube-api-access-fvnsp\") pod \"maas-api-revoke-sub-efaeed0e-bm78q\" (UID: \"05b244d1-56ea-4f24-a7e0-527308b9aa4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"
Oct 29 23:04:11.220037 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.220009    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541dda73-1e06-48f8-84c5-6c295e16286b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03358a9c-m8xmk\" (UID: \"541dda73-1e06-48f8-84c5-6c295e16286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"
Oct 29 23:04:11.220163 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.220099    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr8fs\" (UniqueName: \"kubernetes.io/projected/541dda73-1e06-48f8-84c5-6c295e16286b-kube-api-access-gr8fs\") pod \"maas-api-revoke-sub-03358a9c-m8xmk\" (UID: \"541dda73-1e06-48f8-84c5-6c295e16286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"
Oct 29 23:04:11.220279 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.220251    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541dda73-1e06-48f8-84c5-6c295e16286b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03358a9c-m8xmk\" (UID: \"541dda73-1e06-48f8-84c5-6c295e16286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"
Oct 29 23:04:11.220396 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.220378    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b244d1-56ea-4f24-a7e0-527308b9aa4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efaeed0e-bm78q\" (UID: \"05b244d1-56ea-4f24-a7e0-527308b9aa4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"
Oct 29 23:04:11.320918 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.320874    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541dda73-1e06-48f8-84c5-6c295e16286b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03358a9c-m8xmk\" (UID: \"541dda73-1e06-48f8-84c5-6c295e16286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"
Oct 29 23:04:11.321114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.320953    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b244d1-56ea-4f24-a7e0-527308b9aa4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efaeed0e-bm78q\" (UID: \"05b244d1-56ea-4f24-a7e0-527308b9aa4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"
Oct 29 23:04:11.321114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.320983    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b244d1-56ea-4f24-a7e0-527308b9aa4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efaeed0e-bm78q\" (UID: \"05b244d1-56ea-4f24-a7e0-527308b9aa4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"
Oct 29 23:04:11.321114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.321000    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvnsp\" (UniqueName: \"kubernetes.io/projected/05b244d1-56ea-4f24-a7e0-527308b9aa4a-kube-api-access-fvnsp\") pod \"maas-api-revoke-sub-efaeed0e-bm78q\" (UID: \"05b244d1-56ea-4f24-a7e0-527308b9aa4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"
Oct 29 23:04:11.321114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.321017    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541dda73-1e06-48f8-84c5-6c295e16286b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03358a9c-m8xmk\" (UID: \"541dda73-1e06-48f8-84c5-6c295e16286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"
Oct 29 23:04:11.321114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.321041    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr8fs\" (UniqueName: \"kubernetes.io/projected/541dda73-1e06-48f8-84c5-6c295e16286b-kube-api-access-gr8fs\") pod \"maas-api-revoke-sub-03358a9c-m8xmk\" (UID: \"541dda73-1e06-48f8-84c5-6c295e16286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"
Oct 29 23:04:11.321652 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.321625    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541dda73-1e06-48f8-84c5-6c295e16286b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03358a9c-m8xmk\" (UID: \"541dda73-1e06-48f8-84c5-6c295e16286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"
Oct 29 23:04:11.321739 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.321721    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b244d1-56ea-4f24-a7e0-527308b9aa4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efaeed0e-bm78q\" (UID: \"05b244d1-56ea-4f24-a7e0-527308b9aa4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"
Oct 29 23:04:11.323674 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.323656    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b244d1-56ea-4f24-a7e0-527308b9aa4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efaeed0e-bm78q\" (UID: \"05b244d1-56ea-4f24-a7e0-527308b9aa4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"
Oct 29 23:04:11.323749 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.323705    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541dda73-1e06-48f8-84c5-6c295e16286b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03358a9c-m8xmk\" (UID: \"541dda73-1e06-48f8-84c5-6c295e16286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"
Oct 29 23:04:11.341512 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.341483    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr8fs\" (UniqueName: \"kubernetes.io/projected/541dda73-1e06-48f8-84c5-6c295e16286b-kube-api-access-gr8fs\") pod \"maas-api-revoke-sub-03358a9c-m8xmk\" (UID: \"541dda73-1e06-48f8-84c5-6c295e16286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"
Oct 29 23:04:11.341709 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.341683    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvnsp\" (UniqueName: \"kubernetes.io/projected/05b244d1-56ea-4f24-a7e0-527308b9aa4a-kube-api-access-fvnsp\") pod \"maas-api-revoke-sub-efaeed0e-bm78q\" (UID: \"05b244d1-56ea-4f24-a7e0-527308b9aa4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"
Oct 29 23:04:11.463603 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.463500    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"
Oct 29 23:04:11.488999 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.488958    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"
Oct 29 23:04:11.780247 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.780210    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"]
Oct 29 23:04:11.781192 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:04:11.781159    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541dda73_1e06_48f8_84c5_6c295e16286b.slice/crio-e981f2bda1c8efbad2d6506c6d5933624212069616b39a1ed30e7d71bea84311 WatchSource:0}: Error finding container e981f2bda1c8efbad2d6506c6d5933624212069616b39a1ed30e7d71bea84311: Status 404 returned error can't find the container with id e981f2bda1c8efbad2d6506c6d5933624212069616b39a1ed30e7d71bea84311
Oct 29 23:04:11.817322 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.817288    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"]
Oct 29 23:04:11.818326 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:04:11.818300    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b244d1_56ea_4f24_a7e0_527308b9aa4a.slice/crio-6bc247407050e9b87160853142b030a34d030281ae780465ac591174b30b7763 WatchSource:0}: Error finding container 6bc247407050e9b87160853142b030a34d030281ae780465ac591174b30b7763: Status 404 returned error can't find the container with id 6bc247407050e9b87160853142b030a34d030281ae780465ac591174b30b7763
Oct 29 23:04:11.842968 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.842931    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk" event={"ID":"541dda73-1e06-48f8-84c5-6c295e16286b","Type":"ContainerStarted","Data":"e981f2bda1c8efbad2d6506c6d5933624212069616b39a1ed30e7d71bea84311"}
Oct 29 23:04:11.844359 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:11.844327    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q" event={"ID":"05b244d1-56ea-4f24-a7e0-527308b9aa4a","Type":"ContainerStarted","Data":"6bc247407050e9b87160853142b030a34d030281ae780465ac591174b30b7763"}
Oct 29 23:04:12.110065 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:12.110015    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"]
Oct 29 23:04:12.110279 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:12.110235    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl" podUID="b302c38f-92a7-4243-868b-3c3a04798313" containerName="istio-proxy" containerID="cri-o://407159fd1464fd7c6323f9095cc550fa563acbd84836989fd22dcc5cf62354f7" gracePeriod=30
Oct 29 23:04:12.854564 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:12.854522    4559 generic.go:356] "Generic (PLEG): container finished" podID="541dda73-1e06-48f8-84c5-6c295e16286b" containerID="fd08010233fbf2622e548c1fd977e4fc9f70738656f44a3394a69da865856fac" exitCode=0
Oct 29 23:04:12.855026 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:12.854606    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk" event={"ID":"541dda73-1e06-48f8-84c5-6c295e16286b","Type":"ContainerDied","Data":"fd08010233fbf2622e548c1fd977e4fc9f70738656f44a3394a69da865856fac"}
Oct 29 23:04:12.856421 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:12.856395    4559 generic.go:356] "Generic (PLEG): container finished" podID="05b244d1-56ea-4f24-a7e0-527308b9aa4a" containerID="d72524d63a238374e9c9664bb7337cc7314673978034dfb296279258e13ff548" exitCode=0
Oct 29 23:04:12.856557 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:12.856425    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q" event={"ID":"05b244d1-56ea-4f24-a7e0-527308b9aa4a","Type":"ContainerDied","Data":"d72524d63a238374e9c9664bb7337cc7314673978034dfb296279258e13ff548"}
Oct 29 23:04:13.693636 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:04:13.693598    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:04:14.042376 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.042342    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"
Oct 29 23:04:14.046282 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.046258    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"
Oct 29 23:04:14.046606 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.046588    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05b244d1-56ea-4f24-a7e0-527308b9aa4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b244d1-56ea-4f24-a7e0-527308b9aa4a-maas-api-xks-ca\") pod \"05b244d1-56ea-4f24-a7e0-527308b9aa4a\" (UID: \"05b244d1-56ea-4f24-a7e0-527308b9aa4a\") "
Oct 29 23:04:14.046692 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.046645    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05b244d1-56ea-4f24-a7e0-527308b9aa4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b244d1-56ea-4f24-a7e0-527308b9aa4a-maas-api-openshift-ca\") pod \"05b244d1-56ea-4f24-a7e0-527308b9aa4a\" (UID: \"05b244d1-56ea-4f24-a7e0-527308b9aa4a\") "
Oct 29 23:04:14.046783 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.046770    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05b244d1-56ea-4f24-a7e0-527308b9aa4a-kube-api-access-fvnsp\" (UniqueName: \"kubernetes.io/projected/05b244d1-56ea-4f24-a7e0-527308b9aa4a-kube-api-access-fvnsp\") pod \"05b244d1-56ea-4f24-a7e0-527308b9aa4a\" (UID: \"05b244d1-56ea-4f24-a7e0-527308b9aa4a\") "
Oct 29 23:04:14.047015 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.046991    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05b244d1-56ea-4f24-a7e0-527308b9aa4a-maas-api-openshift-ca" pod "05b244d1-56ea-4f24-a7e0-527308b9aa4a" (UID: "05b244d1-56ea-4f24-a7e0-527308b9aa4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:14.047284 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.047257    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b244d1-56ea-4f24-a7e0-527308b9aa4a-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:14.049009 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.048983    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05b244d1-56ea-4f24-a7e0-527308b9aa4a-maas-api-xks-ca" pod "05b244d1-56ea-4f24-a7e0-527308b9aa4a" (UID: "05b244d1-56ea-4f24-a7e0-527308b9aa4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:14.049154 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.049134    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b244d1-56ea-4f24-a7e0-527308b9aa4a-kube-api-access-fvnsp" pod "05b244d1-56ea-4f24-a7e0-527308b9aa4a" (UID: "05b244d1-56ea-4f24-a7e0-527308b9aa4a"). InnerVolumeSpecName "kube-api-access-fvnsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:14.148495 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.148453    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/541dda73-1e06-48f8-84c5-6c295e16286b-kube-api-access-gr8fs\" (UniqueName: \"kubernetes.io/projected/541dda73-1e06-48f8-84c5-6c295e16286b-kube-api-access-gr8fs\") pod \"541dda73-1e06-48f8-84c5-6c295e16286b\" (UID: \"541dda73-1e06-48f8-84c5-6c295e16286b\") "
Oct 29 23:04:14.148678 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.148503    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/541dda73-1e06-48f8-84c5-6c295e16286b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541dda73-1e06-48f8-84c5-6c295e16286b-maas-api-openshift-ca\") pod \"541dda73-1e06-48f8-84c5-6c295e16286b\" (UID: \"541dda73-1e06-48f8-84c5-6c295e16286b\") "
Oct 29 23:04:14.148678 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.148537    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/541dda73-1e06-48f8-84c5-6c295e16286b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541dda73-1e06-48f8-84c5-6c295e16286b-maas-api-xks-ca\") pod \"541dda73-1e06-48f8-84c5-6c295e16286b\" (UID: \"541dda73-1e06-48f8-84c5-6c295e16286b\") "
Oct 29 23:04:14.148985 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.148964    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvnsp\" (UniqueName: \"kubernetes.io/projected/05b244d1-56ea-4f24-a7e0-527308b9aa4a-kube-api-access-fvnsp\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:14.149089 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.148989    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b244d1-56ea-4f24-a7e0-527308b9aa4a-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:14.149460 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.149384    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/541dda73-1e06-48f8-84c5-6c295e16286b-maas-api-openshift-ca" pod "541dda73-1e06-48f8-84c5-6c295e16286b" (UID: "541dda73-1e06-48f8-84c5-6c295e16286b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:14.151144 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.151114    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/541dda73-1e06-48f8-84c5-6c295e16286b-maas-api-xks-ca" pod "541dda73-1e06-48f8-84c5-6c295e16286b" (UID: "541dda73-1e06-48f8-84c5-6c295e16286b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:14.151548 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.151528    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541dda73-1e06-48f8-84c5-6c295e16286b-kube-api-access-gr8fs" pod "541dda73-1e06-48f8-84c5-6c295e16286b" (UID: "541dda73-1e06-48f8-84c5-6c295e16286b"). InnerVolumeSpecName "kube-api-access-gr8fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:14.250359 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.250311    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr8fs\" (UniqueName: \"kubernetes.io/projected/541dda73-1e06-48f8-84c5-6c295e16286b-kube-api-access-gr8fs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:14.250359 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.250352    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541dda73-1e06-48f8-84c5-6c295e16286b-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:14.250359 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.250366    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541dda73-1e06-48f8-84c5-6c295e16286b-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:14.872471 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.872375    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"
Oct 29 23:04:14.872471 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.872396    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk" event={"ID":"541dda73-1e06-48f8-84c5-6c295e16286b","Type":"ContainerDied","Data":"e981f2bda1c8efbad2d6506c6d5933624212069616b39a1ed30e7d71bea84311"}
Oct 29 23:04:14.872689 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.872488    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e981f2bda1c8efbad2d6506c6d5933624212069616b39a1ed30e7d71bea84311"
Oct 29 23:04:14.874072 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.874051    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"
Oct 29 23:04:14.874072 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.874056    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q" event={"ID":"05b244d1-56ea-4f24-a7e0-527308b9aa4a","Type":"ContainerDied","Data":"6bc247407050e9b87160853142b030a34d030281ae780465ac591174b30b7763"}
Oct 29 23:04:14.874249 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:14.874105    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bc247407050e9b87160853142b030a34d030281ae780465ac591174b30b7763"
Oct 29 23:04:15.307624 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:15.307588    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:04:17.367609 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.367575    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:04:17.384530 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.384454    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-istio-envoy\") pod \"b302c38f-92a7-4243-868b-3c3a04798313\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") "
Oct 29 23:04:17.384530 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.384501    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-workload-certs\") pod \"b302c38f-92a7-4243-868b-3c3a04798313\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") "
Oct 29 23:04:17.384733 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.384536    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b302c38f-92a7-4243-868b-3c3a04798313-kube-api-access-j5kkl\" (UniqueName: \"kubernetes.io/projected/b302c38f-92a7-4243-868b-3c3a04798313-kube-api-access-j5kkl\") pod \"b302c38f-92a7-4243-868b-3c3a04798313\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") "
Oct 29 23:04:17.384733 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.384570    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-workload-socket\") pod \"b302c38f-92a7-4243-868b-3c3a04798313\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") "
Oct 29 23:04:17.384733 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.384587    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-credential-socket\") pod \"b302c38f-92a7-4243-868b-3c3a04798313\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") "
Oct 29 23:04:17.384733 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.384609    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b302c38f-92a7-4243-868b-3c3a04798313-istio-token\" (UniqueName: \"kubernetes.io/projected/b302c38f-92a7-4243-868b-3c3a04798313-istio-token\") pod \"b302c38f-92a7-4243-868b-3c3a04798313\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") "
Oct 29 23:04:17.384733 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.384671    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b302c38f-92a7-4243-868b-3c3a04798313-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b302c38f-92a7-4243-868b-3c3a04798313-istio-podinfo\") pod \"b302c38f-92a7-4243-868b-3c3a04798313\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") "
Oct 29 23:04:17.385075 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.384788    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-istio-data\") pod \"b302c38f-92a7-4243-868b-3c3a04798313\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") "
Oct 29 23:04:17.385075 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.384859    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b302c38f-92a7-4243-868b-3c3a04798313-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b302c38f-92a7-4243-868b-3c3a04798313-istiod-ca-cert\") pod \"b302c38f-92a7-4243-868b-3c3a04798313\" (UID: \"b302c38f-92a7-4243-868b-3c3a04798313\") "
Oct 29 23:04:17.385075 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.384859    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-workload-certs" pod "b302c38f-92a7-4243-868b-3c3a04798313" (UID: "b302c38f-92a7-4243-868b-3c3a04798313"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.385075 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.384912    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-workload-socket" pod "b302c38f-92a7-4243-868b-3c3a04798313" (UID: "b302c38f-92a7-4243-868b-3c3a04798313"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.385075 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.384926    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-credential-socket" pod "b302c38f-92a7-4243-868b-3c3a04798313" (UID: "b302c38f-92a7-4243-868b-3c3a04798313"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.385344 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.385081    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-istio-data" pod "b302c38f-92a7-4243-868b-3c3a04798313" (UID: "b302c38f-92a7-4243-868b-3c3a04798313"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.385344 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.385249    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.385344 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.385269    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.385344 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.385283    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.385344 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.385298    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.385750 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.385723    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b302c38f-92a7-4243-868b-3c3a04798313-istiod-ca-cert" pod "b302c38f-92a7-4243-868b-3c3a04798313" (UID: "b302c38f-92a7-4243-868b-3c3a04798313"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:17.387047 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.387026    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-istio-envoy" pod "b302c38f-92a7-4243-868b-3c3a04798313" (UID: "b302c38f-92a7-4243-868b-3c3a04798313"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.387363 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.387340    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b302c38f-92a7-4243-868b-3c3a04798313-kube-api-access-j5kkl" pod "b302c38f-92a7-4243-868b-3c3a04798313" (UID: "b302c38f-92a7-4243-868b-3c3a04798313"). InnerVolumeSpecName "kube-api-access-j5kkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:17.387624 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.387601    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b302c38f-92a7-4243-868b-3c3a04798313-istio-podinfo" pod "b302c38f-92a7-4243-868b-3c3a04798313" (UID: "b302c38f-92a7-4243-868b-3c3a04798313"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:17.387719 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.387643    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b302c38f-92a7-4243-868b-3c3a04798313-istio-token" pod "b302c38f-92a7-4243-868b-3c3a04798313" (UID: "b302c38f-92a7-4243-868b-3c3a04798313"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:17.486355 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.486313    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5kkl\" (UniqueName: \"kubernetes.io/projected/b302c38f-92a7-4243-868b-3c3a04798313-kube-api-access-j5kkl\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.486355 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.486358    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b302c38f-92a7-4243-868b-3c3a04798313-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.486579 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.486373    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b302c38f-92a7-4243-868b-3c3a04798313-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.486579 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.486386    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b302c38f-92a7-4243-868b-3c3a04798313-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.486579 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.486400    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b302c38f-92a7-4243-868b-3c3a04798313-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.897591 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.897552    4559 generic.go:356] "Generic (PLEG): container finished" podID="b302c38f-92a7-4243-868b-3c3a04798313" containerID="407159fd1464fd7c6323f9095cc550fa563acbd84836989fd22dcc5cf62354f7" exitCode=0
Oct 29 23:04:17.897788 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.897626    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"
Oct 29 23:04:17.897788 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.897632    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl" event={"ID":"b302c38f-92a7-4243-868b-3c3a04798313","Type":"ContainerDied","Data":"407159fd1464fd7c6323f9095cc550fa563acbd84836989fd22dcc5cf62354f7"}
Oct 29 23:04:17.897788 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.897719    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl" event={"ID":"b302c38f-92a7-4243-868b-3c3a04798313","Type":"ContainerDied","Data":"a9a12584d88ce06720eccb96afa92795d24fadc6627812651a368f4631905d2c"}
Oct 29 23:04:17.897788 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.897741    4559 scope.go:122] "RemoveContainer" containerID="407159fd1464fd7c6323f9095cc550fa563acbd84836989fd22dcc5cf62354f7"
Oct 29 23:04:17.912488 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.912466    4559 scope.go:122] "RemoveContainer" containerID="acc784ea146aaa70d158583b457a8fe15e243934948b567fb23c07c92b8c8c42"
Oct 29 23:04:17.922762 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.922734    4559 scope.go:122] "RemoveContainer" containerID="407159fd1464fd7c6323f9095cc550fa563acbd84836989fd22dcc5cf62354f7"
Oct 29 23:04:17.923108 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:04:17.923084    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407159fd1464fd7c6323f9095cc550fa563acbd84836989fd22dcc5cf62354f7\": container with ID starting with 407159fd1464fd7c6323f9095cc550fa563acbd84836989fd22dcc5cf62354f7 not found: ID does not exist" containerID="407159fd1464fd7c6323f9095cc550fa563acbd84836989fd22dcc5cf62354f7"
Oct 29 23:04:17.923163 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.923119    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407159fd1464fd7c6323f9095cc550fa563acbd84836989fd22dcc5cf62354f7"} err="failed to get container status \"407159fd1464fd7c6323f9095cc550fa563acbd84836989fd22dcc5cf62354f7\": rpc error: code = NotFound desc = could not find container \"407159fd1464fd7c6323f9095cc550fa563acbd84836989fd22dcc5cf62354f7\": container with ID starting with 407159fd1464fd7c6323f9095cc550fa563acbd84836989fd22dcc5cf62354f7 not found: ID does not exist"
Oct 29 23:04:17.923163 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.923138    4559 scope.go:122] "RemoveContainer" containerID="acc784ea146aaa70d158583b457a8fe15e243934948b567fb23c07c92b8c8c42"
Oct 29 23:04:17.923404 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:04:17.923384    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc784ea146aaa70d158583b457a8fe15e243934948b567fb23c07c92b8c8c42\": container with ID starting with acc784ea146aaa70d158583b457a8fe15e243934948b567fb23c07c92b8c8c42 not found: ID does not exist" containerID="acc784ea146aaa70d158583b457a8fe15e243934948b567fb23c07c92b8c8c42"
Oct 29 23:04:17.923466 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.923414    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc784ea146aaa70d158583b457a8fe15e243934948b567fb23c07c92b8c8c42"} err="failed to get container status \"acc784ea146aaa70d158583b457a8fe15e243934948b567fb23c07c92b8c8c42\": rpc error: code = NotFound desc = could not find container \"acc784ea146aaa70d158583b457a8fe15e243934948b567fb23c07c92b8c8c42\": container with ID starting with acc784ea146aaa70d158583b457a8fe15e243934948b567fb23c07c92b8c8c42 not found: ID does not exist"
Oct 29 23:04:17.949900 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.949864    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"]
Oct 29 23:04:17.962430 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:17.962396    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-kvnwl"]
Oct 29 23:04:19.227578 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:19.227541    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b302c38f-92a7-4243-868b-3c3a04798313" path="/var/lib/kubelet/pods/b302c38f-92a7-4243-868b-3c3a04798313/volumes"
Oct 29 23:04:21.905724 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:21.905692    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 23:04:21.906803 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:21.906784    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 23:04:23.068934 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:23.068892    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"]
Oct 29 23:04:23.101686 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:23.101644    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"]
Oct 29 23:04:25.296521 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:25.296485    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:04:28.070175 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:28.070095    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz" podUID="6e75cc1b-5e4f-403d-8501-a2dccd8e99a1" containerName="payload-processing" containerID="cri-o://e20204eed6f2e3f319b3c78934d90b7aacc8772c3463fad7962b07328ae03e2b" gracePeriod=25
Oct 29 23:04:28.102730 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:28.102649    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq" podUID="a5f30faf-8940-4ded-b40e-7897f7e4a22b" containerName="payload-pre-processing" containerID="cri-o://d874ee6316165d3f7e06a5d1283e891535a44082c46bb9334a2e64a793044e41" gracePeriod=25
Oct 29 23:04:30.033407 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:30.033363    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"]
Oct 29 23:04:30.046713 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:30.046672    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-22d634-79klt"]
Oct 29 23:04:31.226110 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:31.226067    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a73d88c2-4afe-42d7-824b-c513cd56a432" path="/var/lib/kubelet/pods/a73d88c2-4afe-42d7-824b-c513cd56a432/volumes"
Oct 29 23:04:32.798132 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:32.798047    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq" podUID="a5f30faf-8940-4ded-b40e-7897f7e4a22b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:04:32.798548 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:32.798046    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz" podUID="6e75cc1b-5e4f-403d-8501-a2dccd8e99a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:04:33.404658 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.404618    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"]
Oct 29 23:04:33.404963 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.404935    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v" podUID="f17340ab-f438-4e3f-a5c2-efd46b75b39e" containerName="istio-proxy" containerID="cri-o://e691707e1d997259ca962c77dae0e1753bfc2bae732069668f88d56b3db258a2" gracePeriod=30
Oct 29 23:04:33.889238 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.889149    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"]
Oct 29 23:04:33.889631 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.889599    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b302c38f-92a7-4243-868b-3c3a04798313" containerName="istio-proxy"
Oct 29 23:04:33.889631 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.889610    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b302c38f-92a7-4243-868b-3c3a04798313" containerName="istio-proxy"
Oct 29 23:04:33.889712 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.889650    4559 cpu_manager.go:414] "containerMap: removing container" podUID="541dda73-1e06-48f8-84c5-6c295e16286b" containerName="revoke-keys"
Oct 29 23:04:33.889712 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.889657    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541dda73-1e06-48f8-84c5-6c295e16286b" containerName="revoke-keys"
Oct 29 23:04:33.889712 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.889681    4559 cpu_manager.go:414] "containerMap: removing container" podUID="05b244d1-56ea-4f24-a7e0-527308b9aa4a" containerName="revoke-keys"
Oct 29 23:04:33.889712 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.889686    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05b244d1-56ea-4f24-a7e0-527308b9aa4a" containerName="revoke-keys"
Oct 29 23:04:33.889886 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.889741    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b302c38f-92a7-4243-868b-3c3a04798313" containerName="istio-proxy"
Oct 29 23:04:33.889886 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.889754    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="541dda73-1e06-48f8-84c5-6c295e16286b" containerName="revoke-keys"
Oct 29 23:04:33.889886 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.889761    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="05b244d1-56ea-4f24-a7e0-527308b9aa4a" containerName="revoke-keys"
Oct 29 23:04:33.894849 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.894801    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"
Oct 29 23:04:33.916195 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.916151    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"]
Oct 29 23:04:33.925684 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.925650    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:04:33.925684 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.925658    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:04:33.979727 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.979686    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xbtf\" (UniqueName: \"kubernetes.io/projected/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-kube-api-access-5xbtf\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24\" (UID: \"b0b2951f-7d4f-4ff5-996d-b77fe30e810e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"
Oct 29 23:04:33.979950 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.979834    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24\" (UID: \"b0b2951f-7d4f-4ff5-996d-b77fe30e810e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"
Oct 29 23:04:33.979950 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.979914    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24\" (UID: \"b0b2951f-7d4f-4ff5-996d-b77fe30e810e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"
Oct 29 23:04:33.994622 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.994581    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:04:33.994622 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:33.994607    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:04:34.081104 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:34.081061    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24\" (UID: \"b0b2951f-7d4f-4ff5-996d-b77fe30e810e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"
Oct 29 23:04:34.081327 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:34.081119    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xbtf\" (UniqueName: \"kubernetes.io/projected/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-kube-api-access-5xbtf\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24\" (UID: \"b0b2951f-7d4f-4ff5-996d-b77fe30e810e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"
Oct 29 23:04:34.081327 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:34.081206    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24\" (UID: \"b0b2951f-7d4f-4ff5-996d-b77fe30e810e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"
Oct 29 23:04:34.081848 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:34.081795    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24\" (UID: \"b0b2951f-7d4f-4ff5-996d-b77fe30e810e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"
Oct 29 23:04:34.083995 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:34.083945    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24\" (UID: \"b0b2951f-7d4f-4ff5-996d-b77fe30e810e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"
Oct 29 23:04:34.100963 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:34.100926    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xbtf\" (UniqueName: \"kubernetes.io/projected/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-kube-api-access-5xbtf\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24\" (UID: \"b0b2951f-7d4f-4ff5-996d-b77fe30e810e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"
Oct 29 23:04:34.180907 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:34.180863    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:04:34.208677 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:34.208626    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"
Oct 29 23:04:34.540716 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:34.540678    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"]
Oct 29 23:04:34.542429 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:04:34.542394    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b2951f_7d4f_4ff5_996d_b77fe30e810e.slice/crio-d708b0e42a9e1337688a1da5b87852d8ee7089c67b23dd38be2ff60025295e29 WatchSource:0}: Error finding container d708b0e42a9e1337688a1da5b87852d8ee7089c67b23dd38be2ff60025295e29: Status 404 returned error can't find the container with id d708b0e42a9e1337688a1da5b87852d8ee7089c67b23dd38be2ff60025295e29
Oct 29 23:04:35.037257 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:35.037213    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24" event={"ID":"b0b2951f-7d4f-4ff5-996d-b77fe30e810e","Type":"ContainerStarted","Data":"adb4b19c8cf09a765355d6b735588fb2597b711b26fae8e1cfd3f63f2e0798bb"}
Oct 29 23:04:35.037796 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:35.037286    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24" event={"ID":"b0b2951f-7d4f-4ff5-996d-b77fe30e810e","Type":"ContainerStarted","Data":"d708b0e42a9e1337688a1da5b87852d8ee7089c67b23dd38be2ff60025295e29"}
Oct 29 23:04:35.091895 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:35.091797    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24" podStartSLOduration=2.091776925 podStartE2EDuration="2.091776925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:35.079268738 +0000 UTC m=+3013.990712291" watchObservedRunningTime="2026-10-29 23:04:35.091776925 +0000 UTC m=+3014.003220481"
Oct 29 23:04:36.046207 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:36.046164    4559 generic.go:356] "Generic (PLEG): container finished" podID="b0b2951f-7d4f-4ff5-996d-b77fe30e810e" containerID="adb4b19c8cf09a765355d6b735588fb2597b711b26fae8e1cfd3f63f2e0798bb" exitCode=0
Oct 29 23:04:36.046676 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:36.046222    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24" event={"ID":"b0b2951f-7d4f-4ff5-996d-b77fe30e810e","Type":"ContainerDied","Data":"adb4b19c8cf09a765355d6b735588fb2597b711b26fae8e1cfd3f63f2e0798bb"}
Oct 29 23:04:37.201015 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:37.200982    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"
Oct 29 23:04:37.311444 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:37.311357    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-maas-api-xks-ca\") pod \"b0b2951f-7d4f-4ff5-996d-b77fe30e810e\" (UID: \"b0b2951f-7d4f-4ff5-996d-b77fe30e810e\") "
Oct 29 23:04:37.311444 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:37.311436    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-kube-api-access-5xbtf\" (UniqueName: \"kubernetes.io/projected/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-kube-api-access-5xbtf\") pod \"b0b2951f-7d4f-4ff5-996d-b77fe30e810e\" (UID: \"b0b2951f-7d4f-4ff5-996d-b77fe30e810e\") "
Oct 29 23:04:37.311672 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:37.311463    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-maas-api-openshift-ca\") pod \"b0b2951f-7d4f-4ff5-996d-b77fe30e810e\" (UID: \"b0b2951f-7d4f-4ff5-996d-b77fe30e810e\") "
Oct 29 23:04:37.311936 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:37.311905    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-maas-api-openshift-ca" pod "b0b2951f-7d4f-4ff5-996d-b77fe30e810e" (UID: "b0b2951f-7d4f-4ff5-996d-b77fe30e810e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:37.313775 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:37.313744    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-maas-api-xks-ca" pod "b0b2951f-7d4f-4ff5-996d-b77fe30e810e" (UID: "b0b2951f-7d4f-4ff5-996d-b77fe30e810e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:37.313909 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:37.313751    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-kube-api-access-5xbtf" pod "b0b2951f-7d4f-4ff5-996d-b77fe30e810e" (UID: "b0b2951f-7d4f-4ff5-996d-b77fe30e810e"). InnerVolumeSpecName "kube-api-access-5xbtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:37.412438 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:37.412400    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xbtf\" (UniqueName: \"kubernetes.io/projected/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-kube-api-access-5xbtf\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.412438 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:37.412436    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.412438 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:37.412446    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b2951f-7d4f-4ff5-996d-b77fe30e810e-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.061436 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.061401    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"
Oct 29 23:04:38.061436 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.061417    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24" event={"ID":"b0b2951f-7d4f-4ff5-996d-b77fe30e810e","Type":"ContainerDied","Data":"d708b0e42a9e1337688a1da5b87852d8ee7089c67b23dd38be2ff60025295e29"}
Oct 29 23:04:38.061735 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.061469    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d708b0e42a9e1337688a1da5b87852d8ee7089c67b23dd38be2ff60025295e29"
Oct 29 23:04:38.512043 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.512003    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 29 23:04:38.512619 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.512075    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v" podUID="f17340ab-f438-4e3f-a5c2-efd46b75b39e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 29 23:04:38.676657 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.676583    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 23:04:38.826633 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.826588    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-data\") pod \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") "
Oct 29 23:04:38.826633 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.826638    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-podinfo\") pod \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") "
Oct 29 23:04:38.826963 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.826661    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-token\" (UniqueName: \"kubernetes.io/projected/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-token\") pod \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") "
Oct 29 23:04:38.826963 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.826686    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-workload-socket\") pod \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") "
Oct 29 23:04:38.826963 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.826849    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-credential-socket\") pod \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") "
Oct 29 23:04:38.826963 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.826918    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-workload-certs\") pod \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") "
Oct 29 23:04:38.827199 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.826977    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-envoy\") pod \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") "
Oct 29 23:04:38.827199 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.826997    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-data" pod "f17340ab-f438-4e3f-a5c2-efd46b75b39e" (UID: "f17340ab-f438-4e3f-a5c2-efd46b75b39e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.827199 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.827024    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-workload-socket" pod "f17340ab-f438-4e3f-a5c2-efd46b75b39e" (UID: "f17340ab-f438-4e3f-a5c2-efd46b75b39e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.827199 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.827038    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istiod-ca-cert\") pod \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") "
Oct 29 23:04:38.827199 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.827126    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-credential-socket" pod "f17340ab-f438-4e3f-a5c2-efd46b75b39e" (UID: "f17340ab-f438-4e3f-a5c2-efd46b75b39e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.827199 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.827133    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f17340ab-f438-4e3f-a5c2-efd46b75b39e-kube-api-access-g942v\" (UniqueName: \"kubernetes.io/projected/f17340ab-f438-4e3f-a5c2-efd46b75b39e-kube-api-access-g942v\") pod \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\" (UID: \"f17340ab-f438-4e3f-a5c2-efd46b75b39e\") "
Oct 29 23:04:38.827412 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.827238    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-workload-certs" pod "f17340ab-f438-4e3f-a5c2-efd46b75b39e" (UID: "f17340ab-f438-4e3f-a5c2-efd46b75b39e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.827453 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.827421    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istiod-ca-cert" pod "f17340ab-f438-4e3f-a5c2-efd46b75b39e" (UID: "f17340ab-f438-4e3f-a5c2-efd46b75b39e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:38.827637 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.827616    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.827695 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.827645    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.827695 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.827678    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.827773 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.827695    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.827773 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.827710    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.829525 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.829504    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17340ab-f438-4e3f-a5c2-efd46b75b39e-kube-api-access-g942v" pod "f17340ab-f438-4e3f-a5c2-efd46b75b39e" (UID: "f17340ab-f438-4e3f-a5c2-efd46b75b39e"). InnerVolumeSpecName "kube-api-access-g942v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:38.829612 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.829566    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-envoy" pod "f17340ab-f438-4e3f-a5c2-efd46b75b39e" (UID: "f17340ab-f438-4e3f-a5c2-efd46b75b39e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.829656 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.829635    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-podinfo" pod "f17340ab-f438-4e3f-a5c2-efd46b75b39e" (UID: "f17340ab-f438-4e3f-a5c2-efd46b75b39e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:38.829656 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.829645    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-token" pod "f17340ab-f438-4e3f-a5c2-efd46b75b39e" (UID: "f17340ab-f438-4e3f-a5c2-efd46b75b39e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:38.928732 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.928635    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.928732 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.928673    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.928732 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.928687    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g942v\" (UniqueName: \"kubernetes.io/projected/f17340ab-f438-4e3f-a5c2-efd46b75b39e-kube-api-access-g942v\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.928732 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:38.928699    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f17340ab-f438-4e3f-a5c2-efd46b75b39e-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:39.072634 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:39.072596    4559 generic.go:356] "Generic (PLEG): container finished" podID="f17340ab-f438-4e3f-a5c2-efd46b75b39e" containerID="e691707e1d997259ca962c77dae0e1753bfc2bae732069668f88d56b3db258a2" exitCode=0
Oct 29 23:04:39.072863 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:39.072674    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"
Oct 29 23:04:39.072863 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:39.072685    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v" event={"ID":"f17340ab-f438-4e3f-a5c2-efd46b75b39e","Type":"ContainerDied","Data":"e691707e1d997259ca962c77dae0e1753bfc2bae732069668f88d56b3db258a2"}
Oct 29 23:04:39.072863 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:39.072793    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v" event={"ID":"f17340ab-f438-4e3f-a5c2-efd46b75b39e","Type":"ContainerDied","Data":"306c9174117b96ed7b52007b0e0f522934a987465ffc29e5f18a3d0149c13fad"}
Oct 29 23:04:39.072863 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:39.072844    4559 scope.go:122] "RemoveContainer" containerID="e691707e1d997259ca962c77dae0e1753bfc2bae732069668f88d56b3db258a2"
Oct 29 23:04:39.084491 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:39.084466    4559 scope.go:122] "RemoveContainer" containerID="e691707e1d997259ca962c77dae0e1753bfc2bae732069668f88d56b3db258a2"
Oct 29 23:04:39.084871 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:04:39.084846    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e691707e1d997259ca962c77dae0e1753bfc2bae732069668f88d56b3db258a2\": container with ID starting with e691707e1d997259ca962c77dae0e1753bfc2bae732069668f88d56b3db258a2 not found: ID does not exist" containerID="e691707e1d997259ca962c77dae0e1753bfc2bae732069668f88d56b3db258a2"
Oct 29 23:04:39.084945 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:39.084885    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e691707e1d997259ca962c77dae0e1753bfc2bae732069668f88d56b3db258a2"} err="failed to get container status \"e691707e1d997259ca962c77dae0e1753bfc2bae732069668f88d56b3db258a2\": rpc error: code = NotFound desc = could not find container \"e691707e1d997259ca962c77dae0e1753bfc2bae732069668f88d56b3db258a2\": container with ID starting with e691707e1d997259ca962c77dae0e1753bfc2bae732069668f88d56b3db258a2 not found: ID does not exist"
Oct 29 23:04:39.125623 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:39.125584    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"]
Oct 29 23:04:39.137883 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:39.137846    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-wk57v"]
Oct 29 23:04:39.227538 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:39.227450    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f17340ab-f438-4e3f-a5c2-efd46b75b39e" path="/var/lib/kubelet/pods/f17340ab-f438-4e3f-a5c2-efd46b75b39e/volumes"
Oct 29 23:04:42.798540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:42.798493    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz" podUID="6e75cc1b-5e4f-403d-8501-a2dccd8e99a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:04:42.798540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:42.798522    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq" podUID="a5f30faf-8940-4ded-b40e-7897f7e4a22b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:04:43.693674 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:04:43.693636    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:04:44.335319 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:44.335283    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"]
Oct 29 23:04:44.366547 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:44.366508    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"]
Oct 29 23:04:48.050678 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:48.050643    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"]
Oct 29 23:04:48.063221 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:48.063187    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d68c2c-tfd24"]
Oct 29 23:04:48.365110 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:48.365083    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"
Oct 29 23:04:48.387990 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:48.387967    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"
Oct 29 23:04:48.528550 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:48.528513    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e75cc1b-5e4f-403d-8501-a2dccd8e99a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e75cc1b-5e4f-403d-8501-a2dccd8e99a1-plugins-config-volume\") pod \"6e75cc1b-5e4f-403d-8501-a2dccd8e99a1\" (UID: \"6e75cc1b-5e4f-403d-8501-a2dccd8e99a1\") "
Oct 29 23:04:48.528744 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:48.528629    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f30faf-8940-4ded-b40e-7897f7e4a22b-kube-api-access-s5q6c\" (UniqueName: \"kubernetes.io/projected/a5f30faf-8940-4ded-b40e-7897f7e4a22b-kube-api-access-s5q6c\") pod \"a5f30faf-8940-4ded-b40e-7897f7e4a22b\" (UID: \"a5f30faf-8940-4ded-b40e-7897f7e4a22b\") "
Oct 29 23:04:48.528744 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:48.528669    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e75cc1b-5e4f-403d-8501-a2dccd8e99a1-kube-api-access-vfbj5\" (UniqueName: \"kubernetes.io/projected/6e75cc1b-5e4f-403d-8501-a2dccd8e99a1-kube-api-access-vfbj5\") pod \"6e75cc1b-5e4f-403d-8501-a2dccd8e99a1\" (UID: \"6e75cc1b-5e4f-403d-8501-a2dccd8e99a1\") "
Oct 29 23:04:48.528900 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:48.528788    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f30faf-8940-4ded-b40e-7897f7e4a22b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f30faf-8940-4ded-b40e-7897f7e4a22b-plugins-config-volume\") pod \"a5f30faf-8940-4ded-b40e-7897f7e4a22b\" (UID: \"a5f30faf-8940-4ded-b40e-7897f7e4a22b\") "
Oct 29 23:04:48.529036 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:48.529008    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e75cc1b-5e4f-403d-8501-a2dccd8e99a1-plugins-config-volume" pod "6e75cc1b-5e4f-403d-8501-a2dccd8e99a1" (UID: "6e75cc1b-5e4f-403d-8501-a2dccd8e99a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:48.529203 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:48.529181    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f30faf-8940-4ded-b40e-7897f7e4a22b-plugins-config-volume" pod "a5f30faf-8940-4ded-b40e-7897f7e4a22b" (UID: "a5f30faf-8940-4ded-b40e-7897f7e4a22b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:48.529284 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:48.529230    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e75cc1b-5e4f-403d-8501-a2dccd8e99a1-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:48.531580 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:48.531555    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e75cc1b-5e4f-403d-8501-a2dccd8e99a1-kube-api-access-vfbj5" pod "6e75cc1b-5e4f-403d-8501-a2dccd8e99a1" (UID: "6e75cc1b-5e4f-403d-8501-a2dccd8e99a1"). InnerVolumeSpecName "kube-api-access-vfbj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:48.531694 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:48.531576    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f30faf-8940-4ded-b40e-7897f7e4a22b-kube-api-access-s5q6c" pod "a5f30faf-8940-4ded-b40e-7897f7e4a22b" (UID: "a5f30faf-8940-4ded-b40e-7897f7e4a22b"). InnerVolumeSpecName "kube-api-access-s5q6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:48.630358 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:48.630269    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5q6c\" (UniqueName: \"kubernetes.io/projected/a5f30faf-8940-4ded-b40e-7897f7e4a22b-kube-api-access-s5q6c\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:48.630358 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:48.630301    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfbj5\" (UniqueName: \"kubernetes.io/projected/6e75cc1b-5e4f-403d-8501-a2dccd8e99a1-kube-api-access-vfbj5\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:48.630358 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:48.630312    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f30faf-8940-4ded-b40e-7897f7e4a22b-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:49.143142 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.143104    4559 generic.go:356] "Generic (PLEG): container finished" podID="6e75cc1b-5e4f-403d-8501-a2dccd8e99a1" containerID="e20204eed6f2e3f319b3c78934d90b7aacc8772c3463fad7962b07328ae03e2b" exitCode=0
Oct 29 23:04:49.143685 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.143156    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"
Oct 29 23:04:49.143685 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.143191    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz" event={"ID":"6e75cc1b-5e4f-403d-8501-a2dccd8e99a1","Type":"ContainerDied","Data":"e20204eed6f2e3f319b3c78934d90b7aacc8772c3463fad7962b07328ae03e2b"}
Oct 29 23:04:49.143685 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.143262    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz" event={"ID":"6e75cc1b-5e4f-403d-8501-a2dccd8e99a1","Type":"ContainerDied","Data":"1f271c01c771c990ce58515dde70b5493125bd5958aaee9176efb947ee76b4e8"}
Oct 29 23:04:49.143685 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.143295    4559 scope.go:122] "RemoveContainer" containerID="e20204eed6f2e3f319b3c78934d90b7aacc8772c3463fad7962b07328ae03e2b"
Oct 29 23:04:49.144793 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.144772    4559 generic.go:356] "Generic (PLEG): container finished" podID="a5f30faf-8940-4ded-b40e-7897f7e4a22b" containerID="d874ee6316165d3f7e06a5d1283e891535a44082c46bb9334a2e64a793044e41" exitCode=0
Oct 29 23:04:49.145097 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.144802    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq" event={"ID":"a5f30faf-8940-4ded-b40e-7897f7e4a22b","Type":"ContainerDied","Data":"d874ee6316165d3f7e06a5d1283e891535a44082c46bb9334a2e64a793044e41"}
Oct 29 23:04:49.145097 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.144855    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"
Oct 29 23:04:49.145097 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.144865    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq" event={"ID":"a5f30faf-8940-4ded-b40e-7897f7e4a22b","Type":"ContainerDied","Data":"107de6bf6032c646be9e81ced8f1247982a5fa4e5bde61b1a8350797a7a4a8c3"}
Oct 29 23:04:49.154941 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.154914    4559 scope.go:122] "RemoveContainer" containerID="e20204eed6f2e3f319b3c78934d90b7aacc8772c3463fad7962b07328ae03e2b"
Oct 29 23:04:49.162562 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:04:49.162517    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20204eed6f2e3f319b3c78934d90b7aacc8772c3463fad7962b07328ae03e2b\": container with ID starting with e20204eed6f2e3f319b3c78934d90b7aacc8772c3463fad7962b07328ae03e2b not found: ID does not exist" containerID="e20204eed6f2e3f319b3c78934d90b7aacc8772c3463fad7962b07328ae03e2b"
Oct 29 23:04:49.162717 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.162578    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20204eed6f2e3f319b3c78934d90b7aacc8772c3463fad7962b07328ae03e2b"} err="failed to get container status \"e20204eed6f2e3f319b3c78934d90b7aacc8772c3463fad7962b07328ae03e2b\": rpc error: code = NotFound desc = could not find container \"e20204eed6f2e3f319b3c78934d90b7aacc8772c3463fad7962b07328ae03e2b\": container with ID starting with e20204eed6f2e3f319b3c78934d90b7aacc8772c3463fad7962b07328ae03e2b not found: ID does not exist"
Oct 29 23:04:49.162717 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.162607    4559 scope.go:122] "RemoveContainer" containerID="d874ee6316165d3f7e06a5d1283e891535a44082c46bb9334a2e64a793044e41"
Oct 29 23:04:49.173301 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.173276    4559 scope.go:122] "RemoveContainer" containerID="d874ee6316165d3f7e06a5d1283e891535a44082c46bb9334a2e64a793044e41"
Oct 29 23:04:49.173632 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:04:49.173585    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d874ee6316165d3f7e06a5d1283e891535a44082c46bb9334a2e64a793044e41\": container with ID starting with d874ee6316165d3f7e06a5d1283e891535a44082c46bb9334a2e64a793044e41 not found: ID does not exist" containerID="d874ee6316165d3f7e06a5d1283e891535a44082c46bb9334a2e64a793044e41"
Oct 29 23:04:49.173742 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.173638    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d874ee6316165d3f7e06a5d1283e891535a44082c46bb9334a2e64a793044e41"} err="failed to get container status \"d874ee6316165d3f7e06a5d1283e891535a44082c46bb9334a2e64a793044e41\": rpc error: code = NotFound desc = could not find container \"d874ee6316165d3f7e06a5d1283e891535a44082c46bb9334a2e64a793044e41\": container with ID starting with d874ee6316165d3f7e06a5d1283e891535a44082c46bb9334a2e64a793044e41 not found: ID does not exist"
Oct 29 23:04:49.209022 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.208984    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"]
Oct 29 23:04:49.214275 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.214245    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-af260f-fd896cf88-rb5bz"]
Oct 29 23:04:49.226653 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.226618    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e75cc1b-5e4f-403d-8501-a2dccd8e99a1" path="/var/lib/kubelet/pods/6e75cc1b-5e4f-403d-8501-a2dccd8e99a1/volumes"
Oct 29 23:04:49.227060 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.227047    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b2951f-7d4f-4ff5-996d-b77fe30e810e" path="/var/lib/kubelet/pods/b0b2951f-7d4f-4ff5-996d-b77fe30e810e/volumes"
Oct 29 23:04:49.249248 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.249217    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"]
Oct 29 23:04:49.252528 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.252501    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-af260f-7cfc9454ff-9c5lq"]
Oct 29 23:04:49.336647 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.336590    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q" podUID="bc8573a8-6d74-4f6d-9c05-11a84c15393b" containerName="payload-processing" containerID="cri-o://047bc21fb6737778a4f9f662709dd2163b00903733b0e619fbabdd109635b41f" gracePeriod=25
Oct 29 23:04:49.367529 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:49.367421    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r" podUID="5db0fef2-9d00-44b3-ab17-70678eddbcda" containerName="payload-pre-processing" containerID="cri-o://de39b16af5fa8974e6ed99f94fb4a9d60c2febfb0cf010d36d8efa107d3c6abb" gracePeriod=25
Oct 29 23:04:50.480764 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:50.480717    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:04:50.877889 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:50.877788    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r" podUID="5db0fef2-9d00-44b3-ab17-70678eddbcda" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:04:50.878047 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:50.877881    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q" podUID="bc8573a8-6d74-4f6d-9c05-11a84c15393b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:04:51.227421 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:51.227377    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f30faf-8940-4ded-b40e-7897f7e4a22b" path="/var/lib/kubelet/pods/a5f30faf-8940-4ded-b40e-7897f7e4a22b/volumes"
Oct 29 23:04:54.650867 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:54.650800    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"]
Oct 29 23:04:54.651410 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:54.651151    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq" podUID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" containerName="istio-proxy" containerID="cri-o://bde9e4588b334cc02e0e4f057ef4318001071585fdb3e1717622738a21736b29" gracePeriod=30
Oct 29 23:04:55.316362 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.316323    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"]
Oct 29 23:04:55.316765 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.316752    4559 cpu_manager.go:414] "containerMap: removing container" podUID="f17340ab-f438-4e3f-a5c2-efd46b75b39e" containerName="istio-proxy"
Oct 29 23:04:55.316850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.316767    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f17340ab-f438-4e3f-a5c2-efd46b75b39e" containerName="istio-proxy"
Oct 29 23:04:55.316850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.316792    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a5f30faf-8940-4ded-b40e-7897f7e4a22b" containerName="payload-pre-processing"
Oct 29 23:04:55.316850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.316800    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f30faf-8940-4ded-b40e-7897f7e4a22b" containerName="payload-pre-processing"
Oct 29 23:04:55.316850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.316826    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b302c38f-92a7-4243-868b-3c3a04798313" containerName="istio-proxy"
Oct 29 23:04:55.316850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.316833    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b302c38f-92a7-4243-868b-3c3a04798313" containerName="istio-proxy"
Oct 29 23:04:55.317020 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.316856    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6e75cc1b-5e4f-403d-8501-a2dccd8e99a1" containerName="payload-processing"
Oct 29 23:04:55.317020 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.316863    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e75cc1b-5e4f-403d-8501-a2dccd8e99a1" containerName="payload-processing"
Oct 29 23:04:55.317020 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.316874    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b0b2951f-7d4f-4ff5-996d-b77fe30e810e" containerName="revoke-keys"
Oct 29 23:04:55.317020 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.316879    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b2951f-7d4f-4ff5-996d-b77fe30e810e" containerName="revoke-keys"
Oct 29 23:04:55.317020 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.316939    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f30faf-8940-4ded-b40e-7897f7e4a22b" containerName="payload-pre-processing"
Oct 29 23:04:55.317020 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.316948    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b302c38f-92a7-4243-868b-3c3a04798313" containerName="istio-proxy"
Oct 29 23:04:55.317020 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.316955    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e75cc1b-5e4f-403d-8501-a2dccd8e99a1" containerName="payload-processing"
Oct 29 23:04:55.317020 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.316961    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b2951f-7d4f-4ff5-996d-b77fe30e810e" containerName="revoke-keys"
Oct 29 23:04:55.317020 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.316969    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="f17340ab-f438-4e3f-a5c2-efd46b75b39e" containerName="istio-proxy"
Oct 29 23:04:55.320368 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.320350    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"
Oct 29 23:04:55.340660 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.340617    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"]
Oct 29 23:04:55.375936 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.375909    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:04:55.376137 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.376003    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:04:55.380681 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.380648    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:04:55.380869 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.380718    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:04:55.391348 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.391307    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df8066f-f522-4e34-9273-40b885a63f94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9\" (UID: \"3df8066f-f522-4e34-9273-40b885a63f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"
Oct 29 23:04:55.391569 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.391354    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp92r\" (UniqueName: \"kubernetes.io/projected/3df8066f-f522-4e34-9273-40b885a63f94-kube-api-access-jp92r\") pod \"maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9\" (UID: \"3df8066f-f522-4e34-9273-40b885a63f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"
Oct 29 23:04:55.391569 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.391552    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df8066f-f522-4e34-9273-40b885a63f94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9\" (UID: \"3df8066f-f522-4e34-9273-40b885a63f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"
Oct 29 23:04:55.492364 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.492324    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df8066f-f522-4e34-9273-40b885a63f94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9\" (UID: \"3df8066f-f522-4e34-9273-40b885a63f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"
Oct 29 23:04:55.492364 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.492384    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df8066f-f522-4e34-9273-40b885a63f94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9\" (UID: \"3df8066f-f522-4e34-9273-40b885a63f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"
Oct 29 23:04:55.492644 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.492405    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp92r\" (UniqueName: \"kubernetes.io/projected/3df8066f-f522-4e34-9273-40b885a63f94-kube-api-access-jp92r\") pod \"maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9\" (UID: \"3df8066f-f522-4e34-9273-40b885a63f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"
Oct 29 23:04:55.493366 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.493331    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df8066f-f522-4e34-9273-40b885a63f94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9\" (UID: \"3df8066f-f522-4e34-9273-40b885a63f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"
Oct 29 23:04:55.495071 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.495039    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df8066f-f522-4e34-9273-40b885a63f94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9\" (UID: \"3df8066f-f522-4e34-9273-40b885a63f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"
Oct 29 23:04:55.512706 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.512674    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp92r\" (UniqueName: \"kubernetes.io/projected/3df8066f-f522-4e34-9273-40b885a63f94-kube-api-access-jp92r\") pod \"maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9\" (UID: \"3df8066f-f522-4e34-9273-40b885a63f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"
Oct 29 23:04:55.634224 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.634134    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"
Oct 29 23:04:55.938331 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:55.938304    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"]
Oct 29 23:04:55.939929 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:04:55.939896    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df8066f_f522_4e34_9273_40b885a63f94.slice/crio-86b4b60c0dae223538de7fcf97459cee8d147378c5b97ab6de7fcb858197ebe3 WatchSource:0}: Error finding container 86b4b60c0dae223538de7fcf97459cee8d147378c5b97ab6de7fcb858197ebe3: Status 404 returned error can't find the container with id 86b4b60c0dae223538de7fcf97459cee8d147378c5b97ab6de7fcb858197ebe3
Oct 29 23:04:56.195325 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:56.195230    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9" event={"ID":"3df8066f-f522-4e34-9273-40b885a63f94","Type":"ContainerStarted","Data":"86b4b60c0dae223538de7fcf97459cee8d147378c5b97ab6de7fcb858197ebe3"}
Oct 29 23:04:57.204124 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:57.204090    4559 generic.go:356] "Generic (PLEG): container finished" podID="3df8066f-f522-4e34-9273-40b885a63f94" containerID="51e3a2d5c7b3ab89d362b5110b66f386b47516d2a0588dbc893206e48983a812" exitCode=0
Oct 29 23:04:57.204523 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:57.204191    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9" event={"ID":"3df8066f-f522-4e34-9273-40b885a63f94","Type":"ContainerDied","Data":"51e3a2d5c7b3ab89d362b5110b66f386b47516d2a0588dbc893206e48983a812"}
Oct 29 23:04:58.361935 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:58.361906    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"
Oct 29 23:04:58.416555 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:58.416519    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3df8066f-f522-4e34-9273-40b885a63f94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df8066f-f522-4e34-9273-40b885a63f94-maas-api-xks-ca\") pod \"3df8066f-f522-4e34-9273-40b885a63f94\" (UID: \"3df8066f-f522-4e34-9273-40b885a63f94\") "
Oct 29 23:04:58.416758 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:58.416708    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3df8066f-f522-4e34-9273-40b885a63f94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df8066f-f522-4e34-9273-40b885a63f94-maas-api-openshift-ca\") pod \"3df8066f-f522-4e34-9273-40b885a63f94\" (UID: \"3df8066f-f522-4e34-9273-40b885a63f94\") "
Oct 29 23:04:58.416862 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:58.416829    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df8066f-f522-4e34-9273-40b885a63f94-kube-api-access-jp92r\" (UniqueName: \"kubernetes.io/projected/3df8066f-f522-4e34-9273-40b885a63f94-kube-api-access-jp92r\") pod \"3df8066f-f522-4e34-9273-40b885a63f94\" (UID: \"3df8066f-f522-4e34-9273-40b885a63f94\") "
Oct 29 23:04:58.417108 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:58.417082    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df8066f-f522-4e34-9273-40b885a63f94-maas-api-openshift-ca" pod "3df8066f-f522-4e34-9273-40b885a63f94" (UID: "3df8066f-f522-4e34-9273-40b885a63f94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:58.419367 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:58.419336    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3df8066f-f522-4e34-9273-40b885a63f94-maas-api-xks-ca" pod "3df8066f-f522-4e34-9273-40b885a63f94" (UID: "3df8066f-f522-4e34-9273-40b885a63f94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:58.419747 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:58.419626    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df8066f-f522-4e34-9273-40b885a63f94-kube-api-access-jp92r" pod "3df8066f-f522-4e34-9273-40b885a63f94" (UID: "3df8066f-f522-4e34-9273-40b885a63f94"). InnerVolumeSpecName "kube-api-access-jp92r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:58.518341 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:58.518253    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df8066f-f522-4e34-9273-40b885a63f94-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:58.518341 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:58.518282    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp92r\" (UniqueName: \"kubernetes.io/projected/3df8066f-f522-4e34-9273-40b885a63f94-kube-api-access-jp92r\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:58.518341 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:58.518293    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df8066f-f522-4e34-9273-40b885a63f94-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:59.217617 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.217582    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"
Oct 29 23:04:59.217617 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.217608    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9" event={"ID":"3df8066f-f522-4e34-9273-40b885a63f94","Type":"ContainerDied","Data":"86b4b60c0dae223538de7fcf97459cee8d147378c5b97ab6de7fcb858197ebe3"}
Oct 29 23:04:59.217953 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.217658    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86b4b60c0dae223538de7fcf97459cee8d147378c5b97ab6de7fcb858197ebe3"
Oct 29 23:04:59.577018 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.576927    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:04:59.577018 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.577001    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq" podUID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:04:59.916059 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.916028    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 23:04:59.934891 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.934855    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-token\" (UniqueName: \"kubernetes.io/projected/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-token\") pod \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") "
Oct 29 23:04:59.935049 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.934901    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-credential-socket\") pod \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") "
Oct 29 23:04:59.935049 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.934932    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-envoy\") pod \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") "
Oct 29 23:04:59.935049 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.934967    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-podinfo\") pod \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") "
Oct 29 23:04:59.935049 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.935013    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-workload-certs\") pod \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") "
Oct 29 23:04:59.935049 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.935043    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-data\") pod \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") "
Oct 29 23:04:59.935320 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.935108    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istiod-ca-cert\") pod \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") "
Oct 29 23:04:59.935320 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.935137    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46bbebaf-946c-4aeb-87f3-eac87926c6ab-kube-api-access-fxqnd\" (UniqueName: \"kubernetes.io/projected/46bbebaf-946c-4aeb-87f3-eac87926c6ab-kube-api-access-fxqnd\") pod \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") "
Oct 29 23:04:59.935320 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.935213    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-workload-socket\") pod \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\" (UID: \"46bbebaf-946c-4aeb-87f3-eac87926c6ab\") "
Oct 29 23:04:59.935499 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.935322    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-credential-socket" pod "46bbebaf-946c-4aeb-87f3-eac87926c6ab" (UID: "46bbebaf-946c-4aeb-87f3-eac87926c6ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:59.935499 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.935345    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-workload-certs" pod "46bbebaf-946c-4aeb-87f3-eac87926c6ab" (UID: "46bbebaf-946c-4aeb-87f3-eac87926c6ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:59.935626 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.935608    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:59.935626 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.935614    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-workload-socket" pod "46bbebaf-946c-4aeb-87f3-eac87926c6ab" (UID: "46bbebaf-946c-4aeb-87f3-eac87926c6ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:59.935745 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.935530    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istiod-ca-cert" pod "46bbebaf-946c-4aeb-87f3-eac87926c6ab" (UID: "46bbebaf-946c-4aeb-87f3-eac87926c6ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:59.935745 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.935632    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:04:59.935917 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.935891    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-data" pod "46bbebaf-946c-4aeb-87f3-eac87926c6ab" (UID: "46bbebaf-946c-4aeb-87f3-eac87926c6ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:59.937341 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.937317    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-envoy" pod "46bbebaf-946c-4aeb-87f3-eac87926c6ab" (UID: "46bbebaf-946c-4aeb-87f3-eac87926c6ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:59.937442 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.937357    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-token" pod "46bbebaf-946c-4aeb-87f3-eac87926c6ab" (UID: "46bbebaf-946c-4aeb-87f3-eac87926c6ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:59.937981 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.937958    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-podinfo" pod "46bbebaf-946c-4aeb-87f3-eac87926c6ab" (UID: "46bbebaf-946c-4aeb-87f3-eac87926c6ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:59.937981 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:04:59.937973    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46bbebaf-946c-4aeb-87f3-eac87926c6ab-kube-api-access-fxqnd" pod "46bbebaf-946c-4aeb-87f3-eac87926c6ab" (UID: "46bbebaf-946c-4aeb-87f3-eac87926c6ab"). InnerVolumeSpecName "kube-api-access-fxqnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:00.036331 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.036300    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.036331 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.036328    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.036331 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.036339    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.036613 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.036347    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.036613 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.036354    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.036613 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.036363    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46bbebaf-946c-4aeb-87f3-eac87926c6ab-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.036613 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.036371    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxqnd\" (UniqueName: \"kubernetes.io/projected/46bbebaf-946c-4aeb-87f3-eac87926c6ab-kube-api-access-fxqnd\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.226595 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.226494    4559 generic.go:356] "Generic (PLEG): container finished" podID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" containerID="bde9e4588b334cc02e0e4f057ef4318001071585fdb3e1717622738a21736b29" exitCode=0
Oct 29 23:05:00.226595 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.226550    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"
Oct 29 23:05:00.226870 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.226583    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq" event={"ID":"46bbebaf-946c-4aeb-87f3-eac87926c6ab","Type":"ContainerDied","Data":"bde9e4588b334cc02e0e4f057ef4318001071585fdb3e1717622738a21736b29"}
Oct 29 23:05:00.226870 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.226655    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq" event={"ID":"46bbebaf-946c-4aeb-87f3-eac87926c6ab","Type":"ContainerDied","Data":"dd261ecbaa4632fe6ad70103e51f5b14b0a2a3165799f70f99c79a4552e6c2d6"}
Oct 29 23:05:00.226870 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.226684    4559 scope.go:122] "RemoveContainer" containerID="bde9e4588b334cc02e0e4f057ef4318001071585fdb3e1717622738a21736b29"
Oct 29 23:05:00.240942 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.240827    4559 scope.go:122] "RemoveContainer" containerID="d50058d742aa421d63bc3046bd9424bd60b7726d34a618a5487cd759002c6719"
Oct 29 23:05:00.251746 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.251722    4559 scope.go:122] "RemoveContainer" containerID="bde9e4588b334cc02e0e4f057ef4318001071585fdb3e1717622738a21736b29"
Oct 29 23:05:00.252110 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:05:00.252083    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde9e4588b334cc02e0e4f057ef4318001071585fdb3e1717622738a21736b29\": container with ID starting with bde9e4588b334cc02e0e4f057ef4318001071585fdb3e1717622738a21736b29 not found: ID does not exist" containerID="bde9e4588b334cc02e0e4f057ef4318001071585fdb3e1717622738a21736b29"
Oct 29 23:05:00.252190 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.252126    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde9e4588b334cc02e0e4f057ef4318001071585fdb3e1717622738a21736b29"} err="failed to get container status \"bde9e4588b334cc02e0e4f057ef4318001071585fdb3e1717622738a21736b29\": rpc error: code = NotFound desc = could not find container \"bde9e4588b334cc02e0e4f057ef4318001071585fdb3e1717622738a21736b29\": container with ID starting with bde9e4588b334cc02e0e4f057ef4318001071585fdb3e1717622738a21736b29 not found: ID does not exist"
Oct 29 23:05:00.252190 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.252148    4559 scope.go:122] "RemoveContainer" containerID="d50058d742aa421d63bc3046bd9424bd60b7726d34a618a5487cd759002c6719"
Oct 29 23:05:00.252418 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:05:00.252399    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50058d742aa421d63bc3046bd9424bd60b7726d34a618a5487cd759002c6719\": container with ID starting with d50058d742aa421d63bc3046bd9424bd60b7726d34a618a5487cd759002c6719 not found: ID does not exist" containerID="d50058d742aa421d63bc3046bd9424bd60b7726d34a618a5487cd759002c6719"
Oct 29 23:05:00.252464 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.252427    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50058d742aa421d63bc3046bd9424bd60b7726d34a618a5487cd759002c6719"} err="failed to get container status \"d50058d742aa421d63bc3046bd9424bd60b7726d34a618a5487cd759002c6719\": rpc error: code = NotFound desc = could not find container \"d50058d742aa421d63bc3046bd9424bd60b7726d34a618a5487cd759002c6719\": container with ID starting with d50058d742aa421d63bc3046bd9424bd60b7726d34a618a5487cd759002c6719 not found: ID does not exist"
Oct 29 23:05:00.280748 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.280716    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"]
Oct 29 23:05:00.293209 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.293175    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-6xvfq"]
Oct 29 23:05:00.877879 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.877836    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q" podUID="bc8573a8-6d74-4f6d-9c05-11a84c15393b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:05:00.878327 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:00.877836    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r" podUID="5db0fef2-9d00-44b3-ab17-70678eddbcda" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:05:01.226288 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:01.226253    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" path="/var/lib/kubelet/pods/46bbebaf-946c-4aeb-87f3-eac87926c6ab/volumes"
Oct 29 23:05:05.764255 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:05.764213    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"]
Oct 29 23:05:08.482066 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.482025    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"]
Oct 29 23:05:08.482540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.482448    4559 cpu_manager.go:414] "containerMap: removing container" podUID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" containerName="istio-proxy"
Oct 29 23:05:08.482540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.482459    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" containerName="istio-proxy"
Oct 29 23:05:08.482540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.482485    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3df8066f-f522-4e34-9273-40b885a63f94" containerName="revoke-keys"
Oct 29 23:05:08.482540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.482490    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3df8066f-f522-4e34-9273-40b885a63f94" containerName="revoke-keys"
Oct 29 23:05:08.482683 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.482564    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" containerName="istio-proxy"
Oct 29 23:05:08.482683 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.482575    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3df8066f-f522-4e34-9273-40b885a63f94" containerName="revoke-keys"
Oct 29 23:05:08.487417 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.487395    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"
Oct 29 23:05:08.504605 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.504577    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:05:08.504786 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.504638    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:05:08.504786 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.504655    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:05:08.506603 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.506577    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"]
Oct 29 23:05:08.515839 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.515772    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:05:08.519079 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.519046    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pnwc\" (UniqueName: \"kubernetes.io/projected/c06d48ce-2877-4075-8379-16218f4d51d8-kube-api-access-7pnwc\") pod \"maas-api-revoke-sub-984d3085-m4bcv\" (UID: \"c06d48ce-2877-4075-8379-16218f4d51d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"
Oct 29 23:05:08.519220 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.519163    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c06d48ce-2877-4075-8379-16218f4d51d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-984d3085-m4bcv\" (UID: \"c06d48ce-2877-4075-8379-16218f4d51d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"
Oct 29 23:05:08.519320 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.519296    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c06d48ce-2877-4075-8379-16218f4d51d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-984d3085-m4bcv\" (UID: \"c06d48ce-2877-4075-8379-16218f4d51d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"
Oct 29 23:05:08.620098 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.620051    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c06d48ce-2877-4075-8379-16218f4d51d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-984d3085-m4bcv\" (UID: \"c06d48ce-2877-4075-8379-16218f4d51d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"
Oct 29 23:05:08.620300 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.620131    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pnwc\" (UniqueName: \"kubernetes.io/projected/c06d48ce-2877-4075-8379-16218f4d51d8-kube-api-access-7pnwc\") pod \"maas-api-revoke-sub-984d3085-m4bcv\" (UID: \"c06d48ce-2877-4075-8379-16218f4d51d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"
Oct 29 23:05:08.620300 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.620170    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c06d48ce-2877-4075-8379-16218f4d51d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-984d3085-m4bcv\" (UID: \"c06d48ce-2877-4075-8379-16218f4d51d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"
Oct 29 23:05:08.620962 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.620942    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c06d48ce-2877-4075-8379-16218f4d51d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-984d3085-m4bcv\" (UID: \"c06d48ce-2877-4075-8379-16218f4d51d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"
Oct 29 23:05:08.622783 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.622756    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c06d48ce-2877-4075-8379-16218f4d51d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-984d3085-m4bcv\" (UID: \"c06d48ce-2877-4075-8379-16218f4d51d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"
Oct 29 23:05:08.639554 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.639507    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pnwc\" (UniqueName: \"kubernetes.io/projected/c06d48ce-2877-4075-8379-16218f4d51d8-kube-api-access-7pnwc\") pod \"maas-api-revoke-sub-984d3085-m4bcv\" (UID: \"c06d48ce-2877-4075-8379-16218f4d51d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"
Oct 29 23:05:08.800776 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:08.800674    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"
Oct 29 23:05:09.098521 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:09.098494    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"]
Oct 29 23:05:09.100035 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:05:09.100005    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06d48ce_2877_4075_8379_16218f4d51d8.slice/crio-5dc90fc639c1729d59fb8f565807ac0df4c1c37868490922be3f9b46a364a866 WatchSource:0}: Error finding container 5dc90fc639c1729d59fb8f565807ac0df4c1c37868490922be3f9b46a364a866: Status 404 returned error can't find the container with id 5dc90fc639c1729d59fb8f565807ac0df4c1c37868490922be3f9b46a364a866
Oct 29 23:05:09.284154 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:09.284114    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv" event={"ID":"c06d48ce-2877-4075-8379-16218f4d51d8","Type":"ContainerStarted","Data":"5dc90fc639c1729d59fb8f565807ac0df4c1c37868490922be3f9b46a364a866"}
Oct 29 23:05:09.621641 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:09.621614    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"
Oct 29 23:05:09.669965 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:09.669928    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"
Oct 29 23:05:09.731694 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:09.731655    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc8573a8-6d74-4f6d-9c05-11a84c15393b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8573a8-6d74-4f6d-9c05-11a84c15393b-plugins-config-volume\") pod \"bc8573a8-6d74-4f6d-9c05-11a84c15393b\" (UID: \"bc8573a8-6d74-4f6d-9c05-11a84c15393b\") "
Oct 29 23:05:09.731908 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:09.731733    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc8573a8-6d74-4f6d-9c05-11a84c15393b-kube-api-access-qgmsp\" (UniqueName: \"kubernetes.io/projected/bc8573a8-6d74-4f6d-9c05-11a84c15393b-kube-api-access-qgmsp\") pod \"bc8573a8-6d74-4f6d-9c05-11a84c15393b\" (UID: \"bc8573a8-6d74-4f6d-9c05-11a84c15393b\") "
Oct 29 23:05:09.731908 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:09.731860    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db0fef2-9d00-44b3-ab17-70678eddbcda-kube-api-access-zwddh\" (UniqueName: \"kubernetes.io/projected/5db0fef2-9d00-44b3-ab17-70678eddbcda-kube-api-access-zwddh\") pod \"5db0fef2-9d00-44b3-ab17-70678eddbcda\" (UID: \"5db0fef2-9d00-44b3-ab17-70678eddbcda\") "
Oct 29 23:05:09.732081 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:09.731920    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db0fef2-9d00-44b3-ab17-70678eddbcda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db0fef2-9d00-44b3-ab17-70678eddbcda-plugins-config-volume\") pod \"5db0fef2-9d00-44b3-ab17-70678eddbcda\" (UID: \"5db0fef2-9d00-44b3-ab17-70678eddbcda\") "
Oct 29 23:05:09.732602 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:09.732574    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8573a8-6d74-4f6d-9c05-11a84c15393b-plugins-config-volume" pod "bc8573a8-6d74-4f6d-9c05-11a84c15393b" (UID: "bc8573a8-6d74-4f6d-9c05-11a84c15393b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:09.732722 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:09.732609    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db0fef2-9d00-44b3-ab17-70678eddbcda-plugins-config-volume" pod "5db0fef2-9d00-44b3-ab17-70678eddbcda" (UID: "5db0fef2-9d00-44b3-ab17-70678eddbcda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:09.734556 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:09.734528    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db0fef2-9d00-44b3-ab17-70678eddbcda-kube-api-access-zwddh" pod "5db0fef2-9d00-44b3-ab17-70678eddbcda" (UID: "5db0fef2-9d00-44b3-ab17-70678eddbcda"). InnerVolumeSpecName "kube-api-access-zwddh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:09.734685 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:09.734566    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8573a8-6d74-4f6d-9c05-11a84c15393b-kube-api-access-qgmsp" pod "bc8573a8-6d74-4f6d-9c05-11a84c15393b" (UID: "bc8573a8-6d74-4f6d-9c05-11a84c15393b"). InnerVolumeSpecName "kube-api-access-qgmsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:09.833181 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:09.833139    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8573a8-6d74-4f6d-9c05-11a84c15393b-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:09.833181 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:09.833172    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgmsp\" (UniqueName: \"kubernetes.io/projected/bc8573a8-6d74-4f6d-9c05-11a84c15393b-kube-api-access-qgmsp\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:09.833181 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:09.833182    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwddh\" (UniqueName: \"kubernetes.io/projected/5db0fef2-9d00-44b3-ab17-70678eddbcda-kube-api-access-zwddh\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:09.833181 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:09.833190    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db0fef2-9d00-44b3-ab17-70678eddbcda-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:10.292383 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.292346    4559 generic.go:356] "Generic (PLEG): container finished" podID="c06d48ce-2877-4075-8379-16218f4d51d8" containerID="c5f9c3e3ee4e2df0adfd4c51ff6350e79a939746a16e3aad7fe18c9f4b2ed62f" exitCode=0
Oct 29 23:05:10.292604 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.292429    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv" event={"ID":"c06d48ce-2877-4075-8379-16218f4d51d8","Type":"ContainerDied","Data":"c5f9c3e3ee4e2df0adfd4c51ff6350e79a939746a16e3aad7fe18c9f4b2ed62f"}
Oct 29 23:05:10.293800 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.293777    4559 generic.go:356] "Generic (PLEG): container finished" podID="bc8573a8-6d74-4f6d-9c05-11a84c15393b" containerID="047bc21fb6737778a4f9f662709dd2163b00903733b0e619fbabdd109635b41f" exitCode=0
Oct 29 23:05:10.293930 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.293843    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"
Oct 29 23:05:10.293930 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.293840    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q" event={"ID":"bc8573a8-6d74-4f6d-9c05-11a84c15393b","Type":"ContainerDied","Data":"047bc21fb6737778a4f9f662709dd2163b00903733b0e619fbabdd109635b41f"}
Oct 29 23:05:10.294029 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.293961    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q" event={"ID":"bc8573a8-6d74-4f6d-9c05-11a84c15393b","Type":"ContainerDied","Data":"72961d7e15ef4b8465b4122add4a7a2721231a03c7bd061beca4e61b0715a68f"}
Oct 29 23:05:10.294029 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.293980    4559 scope.go:122] "RemoveContainer" containerID="047bc21fb6737778a4f9f662709dd2163b00903733b0e619fbabdd109635b41f"
Oct 29 23:05:10.295374 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.295286    4559 generic.go:356] "Generic (PLEG): container finished" podID="5db0fef2-9d00-44b3-ab17-70678eddbcda" containerID="de39b16af5fa8974e6ed99f94fb4a9d60c2febfb0cf010d36d8efa107d3c6abb" exitCode=0
Oct 29 23:05:10.295374 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.295327    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r" event={"ID":"5db0fef2-9d00-44b3-ab17-70678eddbcda","Type":"ContainerDied","Data":"de39b16af5fa8974e6ed99f94fb4a9d60c2febfb0cf010d36d8efa107d3c6abb"}
Oct 29 23:05:10.295374 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.295342    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"
Oct 29 23:05:10.295374 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.295354    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r" event={"ID":"5db0fef2-9d00-44b3-ab17-70678eddbcda","Type":"ContainerDied","Data":"36bdae49237718e773805ab3f1f88549e7a35509ec951c95745b3545689a1df0"}
Oct 29 23:05:10.304973 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.304892    4559 scope.go:122] "RemoveContainer" containerID="047bc21fb6737778a4f9f662709dd2163b00903733b0e619fbabdd109635b41f"
Oct 29 23:05:10.305304 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:05:10.305278    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"047bc21fb6737778a4f9f662709dd2163b00903733b0e619fbabdd109635b41f\": container with ID starting with 047bc21fb6737778a4f9f662709dd2163b00903733b0e619fbabdd109635b41f not found: ID does not exist" containerID="047bc21fb6737778a4f9f662709dd2163b00903733b0e619fbabdd109635b41f"
Oct 29 23:05:10.305408 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.305316    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"047bc21fb6737778a4f9f662709dd2163b00903733b0e619fbabdd109635b41f"} err="failed to get container status \"047bc21fb6737778a4f9f662709dd2163b00903733b0e619fbabdd109635b41f\": rpc error: code = NotFound desc = could not find container \"047bc21fb6737778a4f9f662709dd2163b00903733b0e619fbabdd109635b41f\": container with ID starting with 047bc21fb6737778a4f9f662709dd2163b00903733b0e619fbabdd109635b41f not found: ID does not exist"
Oct 29 23:05:10.305408 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.305343    4559 scope.go:122] "RemoveContainer" containerID="de39b16af5fa8974e6ed99f94fb4a9d60c2febfb0cf010d36d8efa107d3c6abb"
Oct 29 23:05:10.321187 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.321152    4559 scope.go:122] "RemoveContainer" containerID="de39b16af5fa8974e6ed99f94fb4a9d60c2febfb0cf010d36d8efa107d3c6abb"
Oct 29 23:05:10.321522 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:05:10.321501    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de39b16af5fa8974e6ed99f94fb4a9d60c2febfb0cf010d36d8efa107d3c6abb\": container with ID starting with de39b16af5fa8974e6ed99f94fb4a9d60c2febfb0cf010d36d8efa107d3c6abb not found: ID does not exist" containerID="de39b16af5fa8974e6ed99f94fb4a9d60c2febfb0cf010d36d8efa107d3c6abb"
Oct 29 23:05:10.321609 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.321530    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de39b16af5fa8974e6ed99f94fb4a9d60c2febfb0cf010d36d8efa107d3c6abb"} err="failed to get container status \"de39b16af5fa8974e6ed99f94fb4a9d60c2febfb0cf010d36d8efa107d3c6abb\": rpc error: code = NotFound desc = could not find container \"de39b16af5fa8974e6ed99f94fb4a9d60c2febfb0cf010d36d8efa107d3c6abb\": container with ID starting with de39b16af5fa8974e6ed99f94fb4a9d60c2febfb0cf010d36d8efa107d3c6abb not found: ID does not exist"
Oct 29 23:05:10.368023 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.367986    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"]
Oct 29 23:05:10.369966 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.369943    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d68c2c-7d66774ccf-wvf5r"]
Oct 29 23:05:10.403272 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.403236    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"]
Oct 29 23:05:10.406761 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.406729    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d68c2c-5c949bdf9b-52x5q"]
Oct 29 23:05:10.764575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:10.764524    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s" podUID="1fb71c3f-8102-4f17-a698-5f8fc204b669" containerName="payload-processing" containerID="cri-o://58191f3da6266783629a1a368bf35ab8410f6b73e0d880d69972b30346daa068" gracePeriod=25
Oct 29 23:05:11.226196 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:11.226155    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db0fef2-9d00-44b3-ab17-70678eddbcda" path="/var/lib/kubelet/pods/5db0fef2-9d00-44b3-ab17-70678eddbcda/volumes"
Oct 29 23:05:11.226538 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:11.226525    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc8573a8-6d74-4f6d-9c05-11a84c15393b" path="/var/lib/kubelet/pods/bc8573a8-6d74-4f6d-9c05-11a84c15393b/volumes"
Oct 29 23:05:11.442913 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:11.442881    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"
Oct 29 23:05:11.549797 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:11.549706    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c06d48ce-2877-4075-8379-16218f4d51d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c06d48ce-2877-4075-8379-16218f4d51d8-maas-api-openshift-ca\") pod \"c06d48ce-2877-4075-8379-16218f4d51d8\" (UID: \"c06d48ce-2877-4075-8379-16218f4d51d8\") "
Oct 29 23:05:11.549984 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:11.549877    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c06d48ce-2877-4075-8379-16218f4d51d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c06d48ce-2877-4075-8379-16218f4d51d8-maas-api-xks-ca\") pod \"c06d48ce-2877-4075-8379-16218f4d51d8\" (UID: \"c06d48ce-2877-4075-8379-16218f4d51d8\") "
Oct 29 23:05:11.549984 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:11.549936    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c06d48ce-2877-4075-8379-16218f4d51d8-kube-api-access-7pnwc\" (UniqueName: \"kubernetes.io/projected/c06d48ce-2877-4075-8379-16218f4d51d8-kube-api-access-7pnwc\") pod \"c06d48ce-2877-4075-8379-16218f4d51d8\" (UID: \"c06d48ce-2877-4075-8379-16218f4d51d8\") "
Oct 29 23:05:11.550113 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:11.550089    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c06d48ce-2877-4075-8379-16218f4d51d8-maas-api-openshift-ca" pod "c06d48ce-2877-4075-8379-16218f4d51d8" (UID: "c06d48ce-2877-4075-8379-16218f4d51d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:11.550219 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:11.550198    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c06d48ce-2877-4075-8379-16218f4d51d8-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.552188 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:11.552155    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c06d48ce-2877-4075-8379-16218f4d51d8-kube-api-access-7pnwc" pod "c06d48ce-2877-4075-8379-16218f4d51d8" (UID: "c06d48ce-2877-4075-8379-16218f4d51d8"). InnerVolumeSpecName "kube-api-access-7pnwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:11.552311 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:11.552163    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c06d48ce-2877-4075-8379-16218f4d51d8-maas-api-xks-ca" pod "c06d48ce-2877-4075-8379-16218f4d51d8" (UID: "c06d48ce-2877-4075-8379-16218f4d51d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:11.650699 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:11.650659    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pnwc\" (UniqueName: \"kubernetes.io/projected/c06d48ce-2877-4075-8379-16218f4d51d8-kube-api-access-7pnwc\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.650699 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:11.650692    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c06d48ce-2877-4075-8379-16218f4d51d8-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:12.028163 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:12.028132    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"]
Oct 29 23:05:12.030151 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:12.030122    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82385d85-jb9r2"]
Oct 29 23:05:12.310382 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:12.310299    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"
Oct 29 23:05:12.310382 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:12.310339    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv" event={"ID":"c06d48ce-2877-4075-8379-16218f4d51d8","Type":"ContainerDied","Data":"5dc90fc639c1729d59fb8f565807ac0df4c1c37868490922be3f9b46a364a866"}
Oct 29 23:05:12.310577 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:12.310387    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dc90fc639c1729d59fb8f565807ac0df4c1c37868490922be3f9b46a364a866"
Oct 29 23:05:13.225995 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:13.225961    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="971d539d-ac5b-4684-8897-eb299ae4c2dc" path="/var/lib/kubelet/pods/971d539d-ac5b-4684-8897-eb299ae4c2dc/volumes"
Oct 29 23:05:13.693150 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:05:13.693112    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:05:20.602381 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:20.602332    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s" podUID="1fb71c3f-8102-4f17-a698-5f8fc204b669" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:05:21.228574 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:21.228542    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:05:24.194282 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:24.194239    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"]
Oct 29 23:05:24.206954 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:24.206900    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fc684b-q9ct9"]
Oct 29 23:05:24.874011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:24.873980    4559 scope.go:122] "RemoveContainer" containerID="c1ab77e66be41dda47cbdce4559e68bc86a434c747279712709b1e58e93af613"
Oct 29 23:05:25.233984 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:25.233943    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3df8066f-f522-4e34-9273-40b885a63f94" path="/var/lib/kubelet/pods/3df8066f-f522-4e34-9273-40b885a63f94/volumes"
Oct 29 23:05:30.601822 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:30.601762    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s" podUID="1fb71c3f-8102-4f17-a698-5f8fc204b669" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:05:30.628416 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:30.628375    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:05:30.765943 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:30.765901    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"]
Oct 29 23:05:30.766168 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:30.766097    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k" podUID="f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" containerName="istio-proxy" containerID="cri-o://5cbae560fc3b9d7a526036a9cfa29a6c4646e9245e1ab0bbf2b8092816f5289c" gracePeriod=30
Oct 29 23:05:31.023756 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:31.023722    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"
Oct 29 23:05:31.142327 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:31.142293    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb71c3f-8102-4f17-a698-5f8fc204b669-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fb71c3f-8102-4f17-a698-5f8fc204b669-plugins-config-volume\") pod \"1fb71c3f-8102-4f17-a698-5f8fc204b669\" (UID: \"1fb71c3f-8102-4f17-a698-5f8fc204b669\") "
Oct 29 23:05:31.142510 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:31.142461    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb71c3f-8102-4f17-a698-5f8fc204b669-kube-api-access-9dmfd\" (UniqueName: \"kubernetes.io/projected/1fb71c3f-8102-4f17-a698-5f8fc204b669-kube-api-access-9dmfd\") pod \"1fb71c3f-8102-4f17-a698-5f8fc204b669\" (UID: \"1fb71c3f-8102-4f17-a698-5f8fc204b669\") "
Oct 29 23:05:31.142713 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:31.142687    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb71c3f-8102-4f17-a698-5f8fc204b669-plugins-config-volume" pod "1fb71c3f-8102-4f17-a698-5f8fc204b669" (UID: "1fb71c3f-8102-4f17-a698-5f8fc204b669"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:31.144889 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:31.144857    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb71c3f-8102-4f17-a698-5f8fc204b669-kube-api-access-9dmfd" pod "1fb71c3f-8102-4f17-a698-5f8fc204b669" (UID: "1fb71c3f-8102-4f17-a698-5f8fc204b669"). InnerVolumeSpecName "kube-api-access-9dmfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:31.243634 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:31.243591    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dmfd\" (UniqueName: \"kubernetes.io/projected/1fb71c3f-8102-4f17-a698-5f8fc204b669-kube-api-access-9dmfd\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:31.243634 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:31.243621    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fb71c3f-8102-4f17-a698-5f8fc204b669-plugins-config-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:31.433650 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:31.433565    4559 generic.go:356] "Generic (PLEG): container finished" podID="1fb71c3f-8102-4f17-a698-5f8fc204b669" containerID="58191f3da6266783629a1a368bf35ab8410f6b73e0d880d69972b30346daa068" exitCode=0
Oct 29 23:05:31.433650 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:31.433621    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"
Oct 29 23:05:31.433894 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:31.433656    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s" event={"ID":"1fb71c3f-8102-4f17-a698-5f8fc204b669","Type":"ContainerDied","Data":"58191f3da6266783629a1a368bf35ab8410f6b73e0d880d69972b30346daa068"}
Oct 29 23:05:31.433894 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:31.433715    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s" event={"ID":"1fb71c3f-8102-4f17-a698-5f8fc204b669","Type":"ContainerDied","Data":"8e5f21bf987dddcf830c2733dd839eb635b0a85b507f5dfed81f4155200b80f8"}
Oct 29 23:05:31.433894 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:31.433739    4559 scope.go:122] "RemoveContainer" containerID="58191f3da6266783629a1a368bf35ab8410f6b73e0d880d69972b30346daa068"
Oct 29 23:05:31.443532 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:31.443502    4559 scope.go:122] "RemoveContainer" containerID="58191f3da6266783629a1a368bf35ab8410f6b73e0d880d69972b30346daa068"
Oct 29 23:05:31.443865 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:05:31.443837    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58191f3da6266783629a1a368bf35ab8410f6b73e0d880d69972b30346daa068\": container with ID starting with 58191f3da6266783629a1a368bf35ab8410f6b73e0d880d69972b30346daa068 not found: ID does not exist" containerID="58191f3da6266783629a1a368bf35ab8410f6b73e0d880d69972b30346daa068"
Oct 29 23:05:31.443944 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:31.443878    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58191f3da6266783629a1a368bf35ab8410f6b73e0d880d69972b30346daa068"} err="failed to get container status \"58191f3da6266783629a1a368bf35ab8410f6b73e0d880d69972b30346daa068\": rpc error: code = NotFound desc = could not find container \"58191f3da6266783629a1a368bf35ab8410f6b73e0d880d69972b30346daa068\": container with ID starting with 58191f3da6266783629a1a368bf35ab8410f6b73e0d880d69972b30346daa068 not found: ID does not exist"
Oct 29 23:05:31.485344 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:31.485307    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"]
Oct 29 23:05:31.489891 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:31.489864    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fc684b-5db88966b-rwg2s"]
Oct 29 23:05:33.226004 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:33.225968    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb71c3f-8102-4f17-a698-5f8fc204b669" path="/var/lib/kubelet/pods/1fb71c3f-8102-4f17-a698-5f8fc204b669/volumes"
Oct 29 23:05:34.696505 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.696465    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c5b76ff8b-zrlxg"]
Oct 29 23:05:34.696968 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.696942    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c06d48ce-2877-4075-8379-16218f4d51d8" containerName="revoke-keys"
Oct 29 23:05:34.696968 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.696961    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c06d48ce-2877-4075-8379-16218f4d51d8" containerName="revoke-keys"
Oct 29 23:05:34.697046 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.696976    4559 cpu_manager.go:414] "containerMap: removing container" podUID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" containerName="istio-proxy"
Oct 29 23:05:34.697046 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.696985    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" containerName="istio-proxy"
Oct 29 23:05:34.697046 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.697005    4559 cpu_manager.go:414] "containerMap: removing container" podUID="1fb71c3f-8102-4f17-a698-5f8fc204b669" containerName="payload-processing"
Oct 29 23:05:34.697046 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.697013    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb71c3f-8102-4f17-a698-5f8fc204b669" containerName="payload-processing"
Oct 29 23:05:34.697182 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.697050    4559 cpu_manager.go:414] "containerMap: removing container" podUID="5db0fef2-9d00-44b3-ab17-70678eddbcda" containerName="payload-pre-processing"
Oct 29 23:05:34.697182 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.697056    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db0fef2-9d00-44b3-ab17-70678eddbcda" containerName="payload-pre-processing"
Oct 29 23:05:34.697182 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.697067    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bc8573a8-6d74-4f6d-9c05-11a84c15393b" containerName="payload-processing"
Oct 29 23:05:34.697182 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.697072    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc8573a8-6d74-4f6d-9c05-11a84c15393b" containerName="payload-processing"
Oct 29 23:05:34.697182 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.697154    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c06d48ce-2877-4075-8379-16218f4d51d8" containerName="revoke-keys"
Oct 29 23:05:34.697182 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.697166    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb71c3f-8102-4f17-a698-5f8fc204b669" containerName="payload-processing"
Oct 29 23:05:34.697182 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.697174    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="46bbebaf-946c-4aeb-87f3-eac87926c6ab" containerName="istio-proxy"
Oct 29 23:05:34.697182 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.697182    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db0fef2-9d00-44b3-ab17-70678eddbcda" containerName="payload-pre-processing"
Oct 29 23:05:34.697431 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.697189    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc8573a8-6d74-4f6d-9c05-11a84c15393b" containerName="payload-processing"
Oct 29 23:05:34.702011 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.701980    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg"
Oct 29 23:05:34.729227 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.729195    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 29 23:05:34.729227 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.729195    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 29 23:05:34.729473 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.729201    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-x42df\""
Oct 29 23:05:34.729473 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.729327    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 29 23:05:34.732019 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.731988    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c5b76ff8b-zrlxg"]
Oct 29 23:05:34.779627 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.779582    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79cp8\" (UniqueName: \"kubernetes.io/projected/348bc31e-9951-48f9-9074-bd955355d927-kube-api-access-79cp8\") pod \"maas-controller-6c5b76ff8b-zrlxg\" (UID: \"348bc31e-9951-48f9-9074-bd955355d927\") " pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg"
Oct 29 23:05:34.779866 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.779732    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/348bc31e-9951-48f9-9074-bd955355d927-webhook-cert\") pod \"maas-controller-6c5b76ff8b-zrlxg\" (UID: \"348bc31e-9951-48f9-9074-bd955355d927\") " pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg"
Oct 29 23:05:34.779866 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.779775    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348bc31e-9951-48f9-9074-bd955355d927-metrics-certs\") pod \"maas-controller-6c5b76ff8b-zrlxg\" (UID: \"348bc31e-9951-48f9-9074-bd955355d927\") " pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg"
Oct 29 23:05:34.881161 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.881118    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79cp8\" (UniqueName: \"kubernetes.io/projected/348bc31e-9951-48f9-9074-bd955355d927-kube-api-access-79cp8\") pod \"maas-controller-6c5b76ff8b-zrlxg\" (UID: \"348bc31e-9951-48f9-9074-bd955355d927\") " pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg"
Oct 29 23:05:34.881387 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.881208    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/348bc31e-9951-48f9-9074-bd955355d927-webhook-cert\") pod \"maas-controller-6c5b76ff8b-zrlxg\" (UID: \"348bc31e-9951-48f9-9074-bd955355d927\") " pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg"
Oct 29 23:05:34.881387 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.881240    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348bc31e-9951-48f9-9074-bd955355d927-metrics-certs\") pod \"maas-controller-6c5b76ff8b-zrlxg\" (UID: \"348bc31e-9951-48f9-9074-bd955355d927\") " pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg"
Oct 29 23:05:34.883893 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.883865    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348bc31e-9951-48f9-9074-bd955355d927-metrics-certs\") pod \"maas-controller-6c5b76ff8b-zrlxg\" (UID: \"348bc31e-9951-48f9-9074-bd955355d927\") " pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg"
Oct 29 23:05:34.884047 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.884025    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/348bc31e-9951-48f9-9074-bd955355d927-webhook-cert\") pod \"maas-controller-6c5b76ff8b-zrlxg\" (UID: \"348bc31e-9951-48f9-9074-bd955355d927\") " pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg"
Oct 29 23:05:34.902107 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:34.902068    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79cp8\" (UniqueName: \"kubernetes.io/projected/348bc31e-9951-48f9-9074-bd955355d927-kube-api-access-79cp8\") pod \"maas-controller-6c5b76ff8b-zrlxg\" (UID: \"348bc31e-9951-48f9-9074-bd955355d927\") " pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg"
Oct 29 23:05:35.014642 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:35.014540    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg"
Oct 29 23:05:35.324375 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:35.324329    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c5b76ff8b-zrlxg"]
Oct 29 23:05:35.326266 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:05:35.326231    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348bc31e_9951_48f9_9074_bd955355d927.slice/crio-56126c0366df2c162273a816450dda528c7926c10cc3f336ae6d5c5143702a6d WatchSource:0}: Error finding container 56126c0366df2c162273a816450dda528c7926c10cc3f336ae6d5c5143702a6d: Status 404 returned error can't find the container with id 56126c0366df2c162273a816450dda528c7926c10cc3f336ae6d5c5143702a6d
Oct 29 23:05:35.465783 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:35.465738    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg" event={"ID":"348bc31e-9951-48f9-9074-bd955355d927","Type":"ContainerStarted","Data":"56126c0366df2c162273a816450dda528c7926c10cc3f336ae6d5c5143702a6d"}
Oct 29 23:05:36.068999 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.068951    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 23:05:36.196243 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.196209    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-credential-socket\") pod \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") "
Oct 29 23:05:36.196463 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.196315    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-podinfo\") pod \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") "
Oct 29 23:05:36.196543 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.196463    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-workload-certs\") pod \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") "
Oct 29 23:05:36.196543 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.196519    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istiod-ca-cert\") pod \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") "
Oct 29 23:05:36.196661 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.196554    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-credential-socket" pod "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" (UID: "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:36.196661 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.196605    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-envoy\") pod \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") "
Oct 29 23:05:36.196770 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.196678    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-workload-socket\") pod \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") "
Oct 29 23:05:36.196770 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.196735    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-token\" (UniqueName: \"kubernetes.io/projected/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-token\") pod \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") "
Oct 29 23:05:36.196770 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.196764    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-kube-api-access-vz2pb\" (UniqueName: \"kubernetes.io/projected/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-kube-api-access-vz2pb\") pod \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") "
Oct 29 23:05:36.196957 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.196782    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-workload-certs" pod "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" (UID: "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:36.196957 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.196837    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-data\") pod \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\" (UID: \"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9\") "
Oct 29 23:05:36.197090 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.197064    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istiod-ca-cert" pod "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" (UID: "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:36.197195 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.197178    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-credential-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.197261 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.197201    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-workload-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.197261 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.197216    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istiod-ca-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.197828 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.197724    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-workload-socket" pod "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" (UID: "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:36.197828 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.197791    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-data" pod "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" (UID: "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:36.199794 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.199767    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-podinfo" pod "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" (UID: "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:05:36.200115 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.200088    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-envoy" pod "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" (UID: "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:36.200647 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.200618    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-kube-api-access-vz2pb" pod "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" (UID: "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9"). InnerVolumeSpecName "kube-api-access-vz2pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:36.218937 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.218891    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-token" pod "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" (UID: "f19ddc8f-507a-4427-91e9-12f4c0fb1ac9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:36.298611 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.298514    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-data\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.298611 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.298555    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-podinfo\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.298611 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.298572    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-envoy\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.298611 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.298585    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-workload-socket\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.298611 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.298601    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-istio-token\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.298611 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.298613    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz2pb\" (UniqueName: \"kubernetes.io/projected/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9-kube-api-access-vz2pb\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.474357 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.474312    4559 generic.go:356] "Generic (PLEG): container finished" podID="f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" containerID="5cbae560fc3b9d7a526036a9cfa29a6c4646e9245e1ab0bbf2b8092816f5289c" exitCode=0
Oct 29 23:05:36.474536 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.474503    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k" event={"ID":"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9","Type":"ContainerDied","Data":"5cbae560fc3b9d7a526036a9cfa29a6c4646e9245e1ab0bbf2b8092816f5289c"}
Oct 29 23:05:36.474588 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.474559    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k" event={"ID":"f19ddc8f-507a-4427-91e9-12f4c0fb1ac9","Type":"ContainerDied","Data":"7266973d44bc1c45556a258d4b7d76118ddcde539ffd4e09bb220a26a7787a15"}
Oct 29 23:05:36.474623 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.474592    4559 scope.go:122] "RemoveContainer" containerID="5cbae560fc3b9d7a526036a9cfa29a6c4646e9245e1ab0bbf2b8092816f5289c"
Oct 29 23:05:36.474759 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.474744    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"
Oct 29 23:05:36.490540 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.490462    4559 scope.go:122] "RemoveContainer" containerID="5cbae560fc3b9d7a526036a9cfa29a6c4646e9245e1ab0bbf2b8092816f5289c"
Oct 29 23:05:36.491004 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:05:36.490979    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cbae560fc3b9d7a526036a9cfa29a6c4646e9245e1ab0bbf2b8092816f5289c\": container with ID starting with 5cbae560fc3b9d7a526036a9cfa29a6c4646e9245e1ab0bbf2b8092816f5289c not found: ID does not exist" containerID="5cbae560fc3b9d7a526036a9cfa29a6c4646e9245e1ab0bbf2b8092816f5289c"
Oct 29 23:05:36.491121 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.491014    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cbae560fc3b9d7a526036a9cfa29a6c4646e9245e1ab0bbf2b8092816f5289c"} err="failed to get container status \"5cbae560fc3b9d7a526036a9cfa29a6c4646e9245e1ab0bbf2b8092816f5289c\": rpc error: code = NotFound desc = could not find container \"5cbae560fc3b9d7a526036a9cfa29a6c4646e9245e1ab0bbf2b8092816f5289c\": container with ID starting with 5cbae560fc3b9d7a526036a9cfa29a6c4646e9245e1ab0bbf2b8092816f5289c not found: ID does not exist"
Oct 29 23:05:36.539962 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.539660    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"]
Oct 29 23:05:36.560065 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:36.559949    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-vnz5k"]
Oct 29 23:05:37.227067 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:37.227038    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" path="/var/lib/kubelet/pods/f19ddc8f-507a-4427-91e9-12f4c0fb1ac9/volumes"
Oct 29 23:05:37.485703 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:37.485593    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg" event={"ID":"348bc31e-9951-48f9-9074-bd955355d927","Type":"ContainerStarted","Data":"554a3f54d5fe31475edebeef2e590d599e43ee4758437a6d5bb7c092e8c13fb2"}
Oct 29 23:05:37.485925 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:37.485885    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg"
Oct 29 23:05:37.528308 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:37.528253    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg" podStartSLOduration=1.712552216 podStartE2EDuration="3.528236161s" totalImagesPullingTime="1.815683945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:37.52656687 +0000 UTC m=+3076.438010425" watchObservedRunningTime="2026-10-29 23:05:37.528236161 +0000 UTC m=+3076.439679715"
Oct 29 23:05:43.693050 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:05:43.693012    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:05:48.496151 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:48.496117    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg"
Oct 29 23:05:53.030926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:53.030893    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"]
Oct 29 23:05:53.042973 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:53.042947    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd4a109c-87srh"]
Oct 29 23:05:53.225685 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:53.225651    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d971a7-9094-47d7-951b-97c08948c7bd" path="/var/lib/kubelet/pods/a7d971a7-9094-47d7-951b-97c08948c7bd/volumes"
Oct 29 23:05:58.999600 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:58.999561    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"]
Oct 29 23:05:59.000051 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:58.999996    4559 cpu_manager.go:414] "containerMap: removing container" podUID="f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" containerName="istio-proxy"
Oct 29 23:05:59.000051 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.000011    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" containerName="istio-proxy"
Oct 29 23:05:59.000137 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.000114    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19ddc8f-507a-4427-91e9-12f4c0fb1ac9" containerName="istio-proxy"
Oct 29 23:05:59.003652 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.003630    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"
Oct 29 23:05:59.026257 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.026214    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:05:59.026257 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.026266    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:05:59.026257 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.026286    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:05:59.026660 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.026390    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:05:59.029553 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.029525    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"]
Oct 29 23:05:59.109629 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.109571    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8jvhs\" (UID: \"7f40ea5c-4c3b-4392-be6a-2c8650593ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"
Oct 29 23:05:59.109629 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.109638    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g7qr\" (UniqueName: \"kubernetes.io/projected/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-kube-api-access-9g7qr\") pod \"maas-api-revoke-sub-85237e45-8jvhs\" (UID: \"7f40ea5c-4c3b-4392-be6a-2c8650593ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"
Oct 29 23:05:59.109982 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.109750    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8jvhs\" (UID: \"7f40ea5c-4c3b-4392-be6a-2c8650593ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"
Oct 29 23:05:59.211183 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.211141    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g7qr\" (UniqueName: \"kubernetes.io/projected/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-kube-api-access-9g7qr\") pod \"maas-api-revoke-sub-85237e45-8jvhs\" (UID: \"7f40ea5c-4c3b-4392-be6a-2c8650593ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"
Oct 29 23:05:59.211376 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.211199    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8jvhs\" (UID: \"7f40ea5c-4c3b-4392-be6a-2c8650593ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"
Oct 29 23:05:59.211376 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.211335    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8jvhs\" (UID: \"7f40ea5c-4c3b-4392-be6a-2c8650593ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"
Oct 29 23:05:59.211975 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.211952    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8jvhs\" (UID: \"7f40ea5c-4c3b-4392-be6a-2c8650593ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"
Oct 29 23:05:59.213877 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.213858    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8jvhs\" (UID: \"7f40ea5c-4c3b-4392-be6a-2c8650593ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"
Oct 29 23:05:59.231746 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.231710    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g7qr\" (UniqueName: \"kubernetes.io/projected/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-kube-api-access-9g7qr\") pod \"maas-api-revoke-sub-85237e45-8jvhs\" (UID: \"7f40ea5c-4c3b-4392-be6a-2c8650593ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"
Oct 29 23:05:59.320318 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.320218    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"
Oct 29 23:05:59.627649 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:05:59.627603    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"]
Oct 29 23:05:59.629230 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:05:59.629201    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f40ea5c_4c3b_4392_be6a_2c8650593ae9.slice/crio-e1d6cf669c3814e2c2865c0eeedb049daf0f31efef419d5d2d75713b93559c43 WatchSource:0}: Error finding container e1d6cf669c3814e2c2865c0eeedb049daf0f31efef419d5d2d75713b93559c43: Status 404 returned error can't find the container with id e1d6cf669c3814e2c2865c0eeedb049daf0f31efef419d5d2d75713b93559c43
Oct 29 23:06:00.633274 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:00.633231    4559 generic.go:356] "Generic (PLEG): container finished" podID="7f40ea5c-4c3b-4392-be6a-2c8650593ae9" containerID="67a253defcd0a5c59307c410eb3382d88c386ff2be866449ca34465e5cc0886b" exitCode=0
Oct 29 23:06:00.633688 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:00.633320    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs" event={"ID":"7f40ea5c-4c3b-4392-be6a-2c8650593ae9","Type":"ContainerDied","Data":"67a253defcd0a5c59307c410eb3382d88c386ff2be866449ca34465e5cc0886b"}
Oct 29 23:06:00.633688 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:00.633386    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs" event={"ID":"7f40ea5c-4c3b-4392-be6a-2c8650593ae9","Type":"ContainerStarted","Data":"e1d6cf669c3814e2c2865c0eeedb049daf0f31efef419d5d2d75713b93559c43"}
Oct 29 23:06:01.776657 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:01.776628    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"
Oct 29 23:06:01.835381 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:01.835342    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-kube-api-access-9g7qr\" (UniqueName: \"kubernetes.io/projected/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-kube-api-access-9g7qr\") pod \"7f40ea5c-4c3b-4392-be6a-2c8650593ae9\" (UID: \"7f40ea5c-4c3b-4392-be6a-2c8650593ae9\") "
Oct 29 23:06:01.835598 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:01.835401    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-maas-api-xks-ca\") pod \"7f40ea5c-4c3b-4392-be6a-2c8650593ae9\" (UID: \"7f40ea5c-4c3b-4392-be6a-2c8650593ae9\") "
Oct 29 23:06:01.835598 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:01.835502    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-maas-api-openshift-ca\") pod \"7f40ea5c-4c3b-4392-be6a-2c8650593ae9\" (UID: \"7f40ea5c-4c3b-4392-be6a-2c8650593ae9\") "
Oct 29 23:06:01.835892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:01.835863    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-maas-api-openshift-ca" pod "7f40ea5c-4c3b-4392-be6a-2c8650593ae9" (UID: "7f40ea5c-4c3b-4392-be6a-2c8650593ae9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:06:01.837669 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:01.837645    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-maas-api-xks-ca" pod "7f40ea5c-4c3b-4392-be6a-2c8650593ae9" (UID: "7f40ea5c-4c3b-4392-be6a-2c8650593ae9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:06:01.837831 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:01.837797    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-kube-api-access-9g7qr" pod "7f40ea5c-4c3b-4392-be6a-2c8650593ae9" (UID: "7f40ea5c-4c3b-4392-be6a-2c8650593ae9"). InnerVolumeSpecName "kube-api-access-9g7qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:06:01.936835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:01.936768    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:06:01.936835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:01.936801    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g7qr\" (UniqueName: \"kubernetes.io/projected/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-kube-api-access-9g7qr\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:06:01.936835 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:01.936837    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f40ea5c-4c3b-4392-be6a-2c8650593ae9-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:06:02.646436 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:02.646360    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"
Oct 29 23:06:02.646436 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:02.646379    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs" event={"ID":"7f40ea5c-4c3b-4392-be6a-2c8650593ae9","Type":"ContainerDied","Data":"e1d6cf669c3814e2c2865c0eeedb049daf0f31efef419d5d2d75713b93559c43"}
Oct 29 23:06:02.646436 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:02.646429    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1d6cf669c3814e2c2865c0eeedb049daf0f31efef419d5d2d75713b93559c43"
Oct 29 23:06:04.554151 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.554110    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"]
Oct 29 23:06:04.554575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.554507    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7f40ea5c-4c3b-4392-be6a-2c8650593ae9" containerName="revoke-keys"
Oct 29 23:06:04.554575 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.554518    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f40ea5c-4c3b-4392-be6a-2c8650593ae9" containerName="revoke-keys"
Oct 29 23:06:04.554654 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.554587    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f40ea5c-4c3b-4392-be6a-2c8650593ae9" containerName="revoke-keys"
Oct 29 23:06:04.557926 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.557907    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.582794 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.582754    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"]
Oct 29 23:06:04.663178 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.663142    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vcd9\" (UniqueName: \"kubernetes.io/projected/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-kube-api-access-2vcd9\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.663178 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.663177    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.663551 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.663204    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.663551 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.663272    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.663551 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.663309    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.663551 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.663332    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.663551 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.663455    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.663551 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.663483    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.663767 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.663591    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.764338 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.764292    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.764338 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.764339    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.764598 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.764387    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.764598 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.764406    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.764598 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.764572    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.764774 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.764672    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vcd9\" (UniqueName: \"kubernetes.io/projected/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-kube-api-access-2vcd9\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.764774 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.764723    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.764774 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.764762    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.764774 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.764761    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.765063 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.764787    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.765063 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.764805    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.765063 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.764888    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.765063 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.765031    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.765499 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.765473    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.767007 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.766986    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.767400 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.767381    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.784056 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.784022    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.784167 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.784144    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vcd9\" (UniqueName: \"kubernetes.io/projected/e0384c7e-3969-4a0e-8e4d-7d9ffa718730-kube-api-access-2vcd9\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-kgx4g\" (UID: \"e0384c7e-3969-4a0e-8e4d-7d9ffa718730\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:04.873541 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:04.873448    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:05.187322 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:05.187296    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"]
Oct 29 23:06:05.189004 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:06:05.188972    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0384c7e_3969_4a0e_8e4d_7d9ffa718730.slice/crio-cceb8c9bcc3404be69114748bd6b017f589cdc78ac81d985fe6a1a50a07eacb2 WatchSource:0}: Error finding container cceb8c9bcc3404be69114748bd6b017f589cdc78ac81d985fe6a1a50a07eacb2: Status 404 returned error can't find the container with id cceb8c9bcc3404be69114748bd6b017f589cdc78ac81d985fe6a1a50a07eacb2
Oct 29 23:06:05.191245 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:05.191211    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:06:05.191341 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:05.191277    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:06:05.191341 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:05.191306    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:06:05.667149 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:05.667100    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g" event={"ID":"e0384c7e-3969-4a0e-8e4d-7d9ffa718730","Type":"ContainerStarted","Data":"38011a589c187266be97898115728b16691dc1f548be9a2931f191af5166a391"}
Oct 29 23:06:05.667670 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:05.667175    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g" event={"ID":"e0384c7e-3969-4a0e-8e4d-7d9ffa718730","Type":"ContainerStarted","Data":"cceb8c9bcc3404be69114748bd6b017f589cdc78ac81d985fe6a1a50a07eacb2"}
Oct 29 23:06:05.722077 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:05.722012    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g" podStartSLOduration=1.7219972289999999 podStartE2EDuration="1.721997229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:05.72057304 +0000 UTC m=+3104.632016594" watchObservedRunningTime="2026-10-29 23:06:05.721997229 +0000 UTC m=+3104.633440774"
Oct 29 23:06:05.874188 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:05.874150    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:06.875271 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:06.875230    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Oct 29 23:06:06.875702 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:06.875288    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g" podUID="e0384c7e-3969-4a0e-8e4d-7d9ffa718730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Oct 29 23:06:07.874120 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:07.874082    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Oct 29 23:06:07.874320 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:07.874150    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g" podUID="e0384c7e-3969-4a0e-8e4d-7d9ffa718730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Oct 29 23:06:08.874027 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:08.873987    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Oct 29 23:06:08.874471 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:08.874055    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g" podUID="e0384c7e-3969-4a0e-8e4d-7d9ffa718730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Oct 29 23:06:09.873757 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:09.873710    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Oct 29 23:06:09.874005 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:09.873780    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g" podUID="e0384c7e-3969-4a0e-8e4d-7d9ffa718730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Oct 29 23:06:10.636175 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:10.636132    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c5b76ff8b-zrlxg"]
Oct 29 23:06:10.637955 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:10.637387    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg" podUID="348bc31e-9951-48f9-9074-bd955355d927" containerName="manager" containerID="cri-o://554a3f54d5fe31475edebeef2e590d599e43ee4758437a6d5bb7c092e8c13fb2" gracePeriod=30
Oct 29 23:06:10.874518 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:10.874480    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Oct 29 23:06:10.874703 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:10.874543    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g" podUID="e0384c7e-3969-4a0e-8e4d-7d9ffa718730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Oct 29 23:06:10.923037 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:10.923011    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg"
Oct 29 23:06:11.018842 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.018781    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/348bc31e-9951-48f9-9074-bd955355d927-kube-api-access-79cp8\" (UniqueName: \"kubernetes.io/projected/348bc31e-9951-48f9-9074-bd955355d927-kube-api-access-79cp8\") pod \"348bc31e-9951-48f9-9074-bd955355d927\" (UID: \"348bc31e-9951-48f9-9074-bd955355d927\") "
Oct 29 23:06:11.019042 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.018901    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/348bc31e-9951-48f9-9074-bd955355d927-webhook-cert\" (UniqueName: \"kubernetes.io/secret/348bc31e-9951-48f9-9074-bd955355d927-webhook-cert\") pod \"348bc31e-9951-48f9-9074-bd955355d927\" (UID: \"348bc31e-9951-48f9-9074-bd955355d927\") "
Oct 29 23:06:11.019042 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.018921    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/348bc31e-9951-48f9-9074-bd955355d927-metrics-certs\" (UniqueName: \"kubernetes.io/secret/348bc31e-9951-48f9-9074-bd955355d927-metrics-certs\") pod \"348bc31e-9951-48f9-9074-bd955355d927\" (UID: \"348bc31e-9951-48f9-9074-bd955355d927\") "
Oct 29 23:06:11.025861 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.021447    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348bc31e-9951-48f9-9074-bd955355d927-webhook-cert" pod "348bc31e-9951-48f9-9074-bd955355d927" (UID: "348bc31e-9951-48f9-9074-bd955355d927"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:06:11.025861 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.021670    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348bc31e-9951-48f9-9074-bd955355d927-metrics-certs" pod "348bc31e-9951-48f9-9074-bd955355d927" (UID: "348bc31e-9951-48f9-9074-bd955355d927"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:06:11.025861 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.021769    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348bc31e-9951-48f9-9074-bd955355d927-kube-api-access-79cp8" pod "348bc31e-9951-48f9-9074-bd955355d927" (UID: "348bc31e-9951-48f9-9074-bd955355d927"). InnerVolumeSpecName "kube-api-access-79cp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:06:11.120307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.120263    4559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/348bc31e-9951-48f9-9074-bd955355d927-webhook-cert\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:06:11.120307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.120300    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348bc31e-9951-48f9-9074-bd955355d927-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:06:11.120307 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.120312    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79cp8\" (UniqueName: \"kubernetes.io/projected/348bc31e-9951-48f9-9074-bd955355d927-kube-api-access-79cp8\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:06:11.714602 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.714544    4559 generic.go:356] "Generic (PLEG): container finished" podID="348bc31e-9951-48f9-9074-bd955355d927" containerID="554a3f54d5fe31475edebeef2e590d599e43ee4758437a6d5bb7c092e8c13fb2" exitCode=0
Oct 29 23:06:11.715146 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.714608    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg" event={"ID":"348bc31e-9951-48f9-9074-bd955355d927","Type":"ContainerDied","Data":"554a3f54d5fe31475edebeef2e590d599e43ee4758437a6d5bb7c092e8c13fb2"}
Oct 29 23:06:11.715146 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.714669    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg"
Oct 29 23:06:11.715146 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.714719    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5b76ff8b-zrlxg" event={"ID":"348bc31e-9951-48f9-9074-bd955355d927","Type":"ContainerDied","Data":"56126c0366df2c162273a816450dda528c7926c10cc3f336ae6d5c5143702a6d"}
Oct 29 23:06:11.715146 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.714753    4559 scope.go:122] "RemoveContainer" containerID="554a3f54d5fe31475edebeef2e590d599e43ee4758437a6d5bb7c092e8c13fb2"
Oct 29 23:06:11.725372 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.725347    4559 scope.go:122] "RemoveContainer" containerID="554a3f54d5fe31475edebeef2e590d599e43ee4758437a6d5bb7c092e8c13fb2"
Oct 29 23:06:11.725721 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:06:11.725699    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"554a3f54d5fe31475edebeef2e590d599e43ee4758437a6d5bb7c092e8c13fb2\": container with ID starting with 554a3f54d5fe31475edebeef2e590d599e43ee4758437a6d5bb7c092e8c13fb2 not found: ID does not exist" containerID="554a3f54d5fe31475edebeef2e590d599e43ee4758437a6d5bb7c092e8c13fb2"
Oct 29 23:06:11.725795 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.725731    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"554a3f54d5fe31475edebeef2e590d599e43ee4758437a6d5bb7c092e8c13fb2"} err="failed to get container status \"554a3f54d5fe31475edebeef2e590d599e43ee4758437a6d5bb7c092e8c13fb2\": rpc error: code = NotFound desc = could not find container \"554a3f54d5fe31475edebeef2e590d599e43ee4758437a6d5bb7c092e8c13fb2\": container with ID starting with 554a3f54d5fe31475edebeef2e590d599e43ee4758437a6d5bb7c092e8c13fb2 not found: ID does not exist"
Oct 29 23:06:11.766956 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.766916    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c5b76ff8b-zrlxg"]
Oct 29 23:06:11.779764 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.779728    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c5b76ff8b-zrlxg"]
Oct 29 23:06:11.874674 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.874632    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Oct 29 23:06:11.874895 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:11.874704    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g" podUID="e0384c7e-3969-4a0e-8e4d-7d9ffa718730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Oct 29 23:06:12.964223 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:12.964182    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c5b76ff8b-rmfgt"]
Oct 29 23:06:12.965250 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:12.965211    4559 cpu_manager.go:414] "containerMap: removing container" podUID="348bc31e-9951-48f9-9074-bd955355d927" containerName="manager"
Oct 29 23:06:12.965250 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:12.965246    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="348bc31e-9951-48f9-9074-bd955355d927" containerName="manager"
Oct 29 23:06:12.965499 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:12.965407    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="348bc31e-9951-48f9-9074-bd955355d927" containerName="manager"
Oct 29 23:06:12.971063 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:12.971033    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5b76ff8b-rmfgt"
Oct 29 23:06:12.981451 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:12.981425    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c5b76ff8b-rmfgt"]
Oct 29 23:06:12.989761 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:12.989729    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 29 23:06:12.989970 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:12.989776    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 29 23:06:12.998927 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:12.998898    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 29 23:06:12.998927 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:12.998900    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-x42df\""
Oct 29 23:06:13.037761 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:13.037721    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvmj9\" (UniqueName: \"kubernetes.io/projected/4d3527ea-abd1-497c-a81e-531b46621769-kube-api-access-jvmj9\") pod \"maas-controller-6c5b76ff8b-rmfgt\" (UID: \"4d3527ea-abd1-497c-a81e-531b46621769\") " pod="opendatahub/maas-controller-6c5b76ff8b-rmfgt"
Oct 29 23:06:13.037969 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:13.037904    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d3527ea-abd1-497c-a81e-531b46621769-webhook-cert\") pod \"maas-controller-6c5b76ff8b-rmfgt\" (UID: \"4d3527ea-abd1-497c-a81e-531b46621769\") " pod="opendatahub/maas-controller-6c5b76ff8b-rmfgt"
Oct 29 23:06:13.038027 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:13.037993    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d3527ea-abd1-497c-a81e-531b46621769-metrics-certs\") pod \"maas-controller-6c5b76ff8b-rmfgt\" (UID: \"4d3527ea-abd1-497c-a81e-531b46621769\") " pod="opendatahub/maas-controller-6c5b76ff8b-rmfgt"
Oct 29 23:06:13.139219 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:13.139175    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d3527ea-abd1-497c-a81e-531b46621769-metrics-certs\") pod \"maas-controller-6c5b76ff8b-rmfgt\" (UID: \"4d3527ea-abd1-497c-a81e-531b46621769\") " pod="opendatahub/maas-controller-6c5b76ff8b-rmfgt"
Oct 29 23:06:13.139445 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:13.139260    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvmj9\" (UniqueName: \"kubernetes.io/projected/4d3527ea-abd1-497c-a81e-531b46621769-kube-api-access-jvmj9\") pod \"maas-controller-6c5b76ff8b-rmfgt\" (UID: \"4d3527ea-abd1-497c-a81e-531b46621769\") " pod="opendatahub/maas-controller-6c5b76ff8b-rmfgt"
Oct 29 23:06:13.139445 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:13.139324    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d3527ea-abd1-497c-a81e-531b46621769-webhook-cert\") pod \"maas-controller-6c5b76ff8b-rmfgt\" (UID: \"4d3527ea-abd1-497c-a81e-531b46621769\") " pod="opendatahub/maas-controller-6c5b76ff8b-rmfgt"
Oct 29 23:06:13.141997 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:13.141967    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d3527ea-abd1-497c-a81e-531b46621769-webhook-cert\") pod \"maas-controller-6c5b76ff8b-rmfgt\" (UID: \"4d3527ea-abd1-497c-a81e-531b46621769\") " pod="opendatahub/maas-controller-6c5b76ff8b-rmfgt"
Oct 29 23:06:13.141997 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:13.141992    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d3527ea-abd1-497c-a81e-531b46621769-metrics-certs\") pod \"maas-controller-6c5b76ff8b-rmfgt\" (UID: \"4d3527ea-abd1-497c-a81e-531b46621769\") " pod="opendatahub/maas-controller-6c5b76ff8b-rmfgt"
Oct 29 23:06:13.159579 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:13.159525    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvmj9\" (UniqueName: \"kubernetes.io/projected/4d3527ea-abd1-497c-a81e-531b46621769-kube-api-access-jvmj9\") pod \"maas-controller-6c5b76ff8b-rmfgt\" (UID: \"4d3527ea-abd1-497c-a81e-531b46621769\") " pod="opendatahub/maas-controller-6c5b76ff8b-rmfgt"
Oct 29 23:06:13.228264 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:13.228177    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="348bc31e-9951-48f9-9074-bd955355d927" path="/var/lib/kubelet/pods/348bc31e-9951-48f9-9074-bd955355d927/volumes"
Oct 29 23:06:13.289702 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:13.289666    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5b76ff8b-rmfgt"
Oct 29 23:06:13.448824 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:13.447524    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:13.448824 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:13.448742    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:13.449357 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:13.449340    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g"
Oct 29 23:06:13.597205 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:13.597177    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c5b76ff8b-rmfgt"]
Oct 29 23:06:13.599187 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:06:13.599149    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3527ea_abd1_497c_a81e_531b46621769.slice/crio-6f728636a105f64c39cd77c9df26650f764ff82929935ccc9cf2ecdbd9323220 WatchSource:0}: Error finding container 6f728636a105f64c39cd77c9df26650f764ff82929935ccc9cf2ecdbd9323220: Status 404 returned error can't find the container with id 6f728636a105f64c39cd77c9df26650f764ff82929935ccc9cf2ecdbd9323220
Oct 29 23:06:13.694253 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:06:13.694211    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:06:13.731957 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:13.731915    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5b76ff8b-rmfgt" event={"ID":"4d3527ea-abd1-497c-a81e-531b46621769","Type":"ContainerStarted","Data":"6f728636a105f64c39cd77c9df26650f764ff82929935ccc9cf2ecdbd9323220"}
Oct 29 23:06:14.739949 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:14.739845    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5b76ff8b-rmfgt" event={"ID":"4d3527ea-abd1-497c-a81e-531b46621769","Type":"ContainerStarted","Data":"1d6fa1df5b44f34163bf3633d3bb33203ab5b7aa78c3e5b4c679e96e4050fb6c"}
Oct 29 23:06:14.740433 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:14.740071    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c5b76ff8b-rmfgt"
Oct 29 23:06:14.780886 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:14.780835    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6c5b76ff8b-rmfgt" podStartSLOduration=2.340966792 podStartE2EDuration="2.78079638s" totalImagesPullingTime="439.829588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:14.778458719 +0000 UTC m=+3113.689902273" watchObservedRunningTime="2026-10-29 23:06:14.78079638 +0000 UTC m=+3113.692239979"
Oct 29 23:06:24.925360 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:24.925323    4559 scope.go:122] "RemoveContainer" containerID="329aa3e1dd370ee4fe4fc735d471f0ea0f429900501b70c0e34c31156d06da49"
Oct 29 23:06:24.935650 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:24.935624    4559 scope.go:122] "RemoveContainer" containerID="2d85e0aecfa1ea2b053a2f64a9aa0468b593dc798e1a1f087e3b48cd7b7b51ca"
Oct 29 23:06:24.944552 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:24.944532    4559 scope.go:122] "RemoveContainer" containerID="21775d168f0fbd4bdb4efafb9d854f4eea1ea3da3fa9949d2e4bd6ee0d4dbb94"
Oct 29 23:06:24.953674 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:24.953647    4559 scope.go:122] "RemoveContainer" containerID="c4e77de649c60c71a3675ec5d6c28f4ab83d89f9eb7c3ae43e4c239cebd15a12"
Oct 29 23:06:24.962512 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:24.962484    4559 scope.go:122] "RemoveContainer" containerID="e5576a935a4eb7ba470c96a549566b2650797d5d73eb14bc44219803d43266e8"
Oct 29 23:06:25.029385 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:25.029350    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"]
Oct 29 23:06:25.042096 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:25.042064    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9f821fe-ll528"]
Oct 29 23:06:25.044338 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:25.044310    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"]
Oct 29 23:06:25.046059 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:25.046030    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f890f139-n5qpl"]
Oct 29 23:06:25.225749 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:25.225659    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="989e4f47-30aa-44b7-bb2c-052ce6760a82" path="/var/lib/kubelet/pods/989e4f47-30aa-44b7-bb2c-052ce6760a82/volumes"
Oct 29 23:06:25.226141 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:25.226126    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5dda999-f738-4ddc-bdc2-02a322c38723" path="/var/lib/kubelet/pods/f5dda999-f738-4ddc-bdc2-02a322c38723/volumes"
Oct 29 23:06:25.748190 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:25.748160    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c5b76ff8b-rmfgt"
Oct 29 23:06:36.035987 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:36.035951    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"]
Oct 29 23:06:36.048428 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:36.048391    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f6b42db-rq9l2"]
Oct 29 23:06:37.225819 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:37.225773    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4940d541-edda-4219-8e75-2d3702b5c3c6" path="/var/lib/kubelet/pods/4940d541-edda-4219-8e75-2d3702b5c3c6/volumes"
Oct 29 23:06:41.609195 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:41.609154    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:06:43.722800 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:06:43.722762    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:06:47.417155 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:47.417121    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"]
Oct 29 23:06:47.422686 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:47.422668    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"
Oct 29 23:06:47.452119 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:47.452087    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:06:47.452302 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:47.452144    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:06:47.452302 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:47.452150    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:06:47.452302 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:47.452186    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:06:47.454255 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:47.454231    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"]
Oct 29 23:06:47.501280 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:47.501236    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q4pl\" (UniqueName: \"kubernetes.io/projected/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-kube-api-access-5q4pl\") pod \"maas-api-revoke-sub-c1e271a9-lc9ct\" (UID: \"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"
Oct 29 23:06:47.501475 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:47.501387    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-lc9ct\" (UID: \"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"
Oct 29 23:06:47.501475 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:47.501443    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-lc9ct\" (UID: \"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"
Oct 29 23:06:47.602351 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:47.602309    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-lc9ct\" (UID: \"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"
Oct 29 23:06:47.602351 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:47.602355    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-lc9ct\" (UID: \"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"
Oct 29 23:06:47.602587 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:47.602422    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q4pl\" (UniqueName: \"kubernetes.io/projected/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-kube-api-access-5q4pl\") pod \"maas-api-revoke-sub-c1e271a9-lc9ct\" (UID: \"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"
Oct 29 23:06:47.610541 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:47.603993    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-lc9ct\" (UID: \"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"
Oct 29 23:06:47.612286 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:47.612247    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-lc9ct\" (UID: \"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"
Oct 29 23:06:47.622585 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:47.622553    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q4pl\" (UniqueName: \"kubernetes.io/projected/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-kube-api-access-5q4pl\") pod \"maas-api-revoke-sub-c1e271a9-lc9ct\" (UID: \"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"
Oct 29 23:06:47.736017 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:47.735933    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"
Oct 29 23:06:48.032519 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:48.032494    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"]
Oct 29 23:06:48.034419 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:06:48.034385    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcf5042_3ee9_4ffd_bad7_cfe83163f2d4.slice/crio-692d86aa073cb33c1da1c69a3d74ccd4a833ead13e9edb776c6a99d2eab707d9 WatchSource:0}: Error finding container 692d86aa073cb33c1da1c69a3d74ccd4a833ead13e9edb776c6a99d2eab707d9: Status 404 returned error can't find the container with id 692d86aa073cb33c1da1c69a3d74ccd4a833ead13e9edb776c6a99d2eab707d9
Oct 29 23:06:48.949713 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:48.947260    4559 generic.go:356] "Generic (PLEG): container finished" podID="cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4" containerID="403cb63076b6fe1fb65d7e61042c58cffb76fbf04de00e95acdf0f7d1925cf9f" exitCode=0
Oct 29 23:06:48.949713 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:48.947347    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct" event={"ID":"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4","Type":"ContainerDied","Data":"403cb63076b6fe1fb65d7e61042c58cffb76fbf04de00e95acdf0f7d1925cf9f"}
Oct 29 23:06:48.949713 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:48.947409    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct" event={"ID":"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4","Type":"ContainerStarted","Data":"692d86aa073cb33c1da1c69a3d74ccd4a833ead13e9edb776c6a99d2eab707d9"}
Oct 29 23:06:50.087535 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:50.087501    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"
Oct 29 23:06:50.127080 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:50.127041    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-maas-api-openshift-ca\") pod \"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4\" (UID: \"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4\") "
Oct 29 23:06:50.127319 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:50.127136    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-maas-api-xks-ca\") pod \"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4\" (UID: \"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4\") "
Oct 29 23:06:50.127401 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:50.127296    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-kube-api-access-5q4pl\" (UniqueName: \"kubernetes.io/projected/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-kube-api-access-5q4pl\") pod \"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4\" (UID: \"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4\") "
Oct 29 23:06:50.127587 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:50.127559    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-maas-api-openshift-ca" pod "cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4" (UID: "cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:06:50.127881 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:50.127860    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:06:50.130090 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:50.130055    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-kube-api-access-5q4pl" pod "cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4" (UID: "cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4"). InnerVolumeSpecName "kube-api-access-5q4pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:06:50.130236 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:50.130133    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-maas-api-xks-ca" pod "cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4" (UID: "cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:06:50.229308 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:50.229203    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:06:50.229308 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:50.229243    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q4pl\" (UniqueName: \"kubernetes.io/projected/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4-kube-api-access-5q4pl\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:06:50.960303 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:50.960273    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"
Oct 29 23:06:50.960491 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:50.960307    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct" event={"ID":"cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4","Type":"ContainerDied","Data":"692d86aa073cb33c1da1c69a3d74ccd4a833ead13e9edb776c6a99d2eab707d9"}
Oct 29 23:06:50.960491 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:50.960357    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="692d86aa073cb33c1da1c69a3d74ccd4a833ead13e9edb776c6a99d2eab707d9"
Oct 29 23:06:58.673639 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:06:58.673596    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:07:03.071454 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:03.071418    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:07:04.709021 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.708975    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"]
Oct 29 23:07:04.709638 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.709615    4559 cpu_manager.go:414] "containerMap: removing container" podUID="cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4" containerName="revoke-keys"
Oct 29 23:07:04.709736 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.709641    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4" containerName="revoke-keys"
Oct 29 23:07:04.709834 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.709781    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4" containerName="revoke-keys"
Oct 29 23:07:04.713108 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.713076    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.758875 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.752718    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"]
Oct 29 23:07:04.773729 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.773680    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5716b06a-0198-4647-884c-3251a6a4b63e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.777358 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.777323    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5716b06a-0198-4647-884c-3251a6a4b63e-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.777677 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.777652    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5716b06a-0198-4647-884c-3251a6a4b63e-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.777893 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.777856    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5716b06a-0198-4647-884c-3251a6a4b63e-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.778106 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.778088    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5716b06a-0198-4647-884c-3251a6a4b63e-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.778337 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.778317    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5716b06a-0198-4647-884c-3251a6a4b63e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.778492 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.778474    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5716b06a-0198-4647-884c-3251a6a4b63e-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.778689 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.778655    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh9q6\" (UniqueName: \"kubernetes.io/projected/5716b06a-0198-4647-884c-3251a6a4b63e-kube-api-access-gh9q6\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.778890 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.778857    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5716b06a-0198-4647-884c-3251a6a4b63e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.879927 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.879880    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5716b06a-0198-4647-884c-3251a6a4b63e-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.880569 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.880533    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5716b06a-0198-4647-884c-3251a6a4b63e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.880783 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.880763    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5716b06a-0198-4647-884c-3251a6a4b63e-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.880974 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.880590    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5716b06a-0198-4647-884c-3251a6a4b63e-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.881080 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.880968    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9q6\" (UniqueName: \"kubernetes.io/projected/5716b06a-0198-4647-884c-3251a6a4b63e-kube-api-access-gh9q6\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.881213 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.881198    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5716b06a-0198-4647-884c-3251a6a4b63e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.881336 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.881320    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5716b06a-0198-4647-884c-3251a6a4b63e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.881558 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.881537    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5716b06a-0198-4647-884c-3251a6a4b63e-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.881708 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.881691    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5716b06a-0198-4647-884c-3251a6a4b63e-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.881879 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.881857    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5716b06a-0198-4647-884c-3251a6a4b63e-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.882317 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.882254    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5716b06a-0198-4647-884c-3251a6a4b63e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.882647 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.882625    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5716b06a-0198-4647-884c-3251a6a4b63e-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.882783 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.881263    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5716b06a-0198-4647-884c-3251a6a4b63e-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.883166 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.883145    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5716b06a-0198-4647-884c-3251a6a4b63e-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.893901 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.884665    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5716b06a-0198-4647-884c-3251a6a4b63e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.893901 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.891691    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5716b06a-0198-4647-884c-3251a6a4b63e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.902207 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.902175    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh9q6\" (UniqueName: \"kubernetes.io/projected/5716b06a-0198-4647-884c-3251a6a4b63e-kube-api-access-gh9q6\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:04.902568 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:04.902548    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5716b06a-0198-4647-884c-3251a6a4b63e-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9g4hg\" (UID: \"5716b06a-0198-4647-884c-3251a6a4b63e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:05.027052 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:05.026958    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:05.388747 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:07:05.388717    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5716b06a_0198_4647_884c_3251a6a4b63e.slice/crio-1b5e6ecbe12c2320ff64ea39f2c8850120657339d4f8565d8e67f5f3a9c82362 WatchSource:0}: Error finding container 1b5e6ecbe12c2320ff64ea39f2c8850120657339d4f8565d8e67f5f3a9c82362: Status 404 returned error can't find the container with id 1b5e6ecbe12c2320ff64ea39f2c8850120657339d4f8565d8e67f5f3a9c82362
Oct 29 23:07:05.392507 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:05.391718    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:07:05.392507 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:05.391847    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:07:05.392507 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:05.391905    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:07:05.392507 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:05.392249    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"]
Oct 29 23:07:06.126769 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:06.126726    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg" event={"ID":"5716b06a-0198-4647-884c-3251a6a4b63e","Type":"ContainerStarted","Data":"b585970aab0232576710213066ee25e4c703232688e669fa1f9132a1a44c1992"}
Oct 29 23:07:06.127343 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:06.126784    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg" event={"ID":"5716b06a-0198-4647-884c-3251a6a4b63e","Type":"ContainerStarted","Data":"1b5e6ecbe12c2320ff64ea39f2c8850120657339d4f8565d8e67f5f3a9c82362"}
Oct 29 23:07:06.195169 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:06.195107    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg" podStartSLOduration=2.195092064 podStartE2EDuration="2.195092064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:06.181383168 +0000 UTC m=+3165.092826719" watchObservedRunningTime="2026-10-29 23:07:06.195092064 +0000 UTC m=+3165.106535617"
Oct 29 23:07:07.028480 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:07.028442    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:07.029957 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:07.029933    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body=
Oct 29 23:07:07.030046 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:07.029985    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg" podUID="5716b06a-0198-4647-884c-3251a6a4b63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused"
Oct 29 23:07:08.028347 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.028304    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body=
Oct 29 23:07:08.028891 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.028365    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg" podUID="5716b06a-0198-4647-884c-3251a6a4b63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused"
Oct 29 23:07:08.545838 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.544531    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"]
Oct 29 23:07:08.556790 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.556757    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"
Oct 29 23:07:08.571447 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.571414    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"]
Oct 29 23:07:08.572731 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.572704    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:07:08.579691 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.579656    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:07:08.579929 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.579698    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:07:08.579929 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.579859    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:07:08.653734 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.653391    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz5bn\" (UniqueName: \"kubernetes.io/projected/3a4d57b5-7fd3-470c-8716-dc3053877b4e-kube-api-access-zz5bn\") pod \"maas-api-revoke-sub-2e1f5b5f-dp4d7\" (UID: \"3a4d57b5-7fd3-470c-8716-dc3053877b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"
Oct 29 23:07:08.653734 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.653494    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a4d57b5-7fd3-470c-8716-dc3053877b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dp4d7\" (UID: \"3a4d57b5-7fd3-470c-8716-dc3053877b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"
Oct 29 23:07:08.653734 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.653585    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a4d57b5-7fd3-470c-8716-dc3053877b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dp4d7\" (UID: \"3a4d57b5-7fd3-470c-8716-dc3053877b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"
Oct 29 23:07:08.758293 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.758258    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz5bn\" (UniqueName: \"kubernetes.io/projected/3a4d57b5-7fd3-470c-8716-dc3053877b4e-kube-api-access-zz5bn\") pod \"maas-api-revoke-sub-2e1f5b5f-dp4d7\" (UID: \"3a4d57b5-7fd3-470c-8716-dc3053877b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"
Oct 29 23:07:08.758490 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.758320    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a4d57b5-7fd3-470c-8716-dc3053877b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dp4d7\" (UID: \"3a4d57b5-7fd3-470c-8716-dc3053877b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"
Oct 29 23:07:08.758490 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.758375    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a4d57b5-7fd3-470c-8716-dc3053877b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dp4d7\" (UID: \"3a4d57b5-7fd3-470c-8716-dc3053877b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"
Oct 29 23:07:08.759086 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.759059    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a4d57b5-7fd3-470c-8716-dc3053877b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dp4d7\" (UID: \"3a4d57b5-7fd3-470c-8716-dc3053877b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"
Oct 29 23:07:08.761098 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.761065    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a4d57b5-7fd3-470c-8716-dc3053877b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dp4d7\" (UID: \"3a4d57b5-7fd3-470c-8716-dc3053877b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"
Oct 29 23:07:08.778577 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.778545    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz5bn\" (UniqueName: \"kubernetes.io/projected/3a4d57b5-7fd3-470c-8716-dc3053877b4e-kube-api-access-zz5bn\") pod \"maas-api-revoke-sub-2e1f5b5f-dp4d7\" (UID: \"3a4d57b5-7fd3-470c-8716-dc3053877b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"
Oct 29 23:07:08.874877 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:08.874784    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"
Oct 29 23:07:09.027420 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:09.027384    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body=
Oct 29 23:07:09.027651 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:09.027441    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg" podUID="5716b06a-0198-4647-884c-3251a6a4b63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused"
Oct 29 23:07:09.184531 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:09.184504    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"]
Oct 29 23:07:09.186264 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:07:09.186240    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a4d57b5_7fd3_470c_8716_dc3053877b4e.slice/crio-1bc2182a53c3312671d24a9ac9453ca6cd02cdc11469f81972bc01ac3962818c WatchSource:0}: Error finding container 1bc2182a53c3312671d24a9ac9453ca6cd02cdc11469f81972bc01ac3962818c: Status 404 returned error can't find the container with id 1bc2182a53c3312671d24a9ac9453ca6cd02cdc11469f81972bc01ac3962818c
Oct 29 23:07:10.027823 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:10.027767    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body=
Oct 29 23:07:10.028022 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:10.027836    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg" podUID="5716b06a-0198-4647-884c-3251a6a4b63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused"
Oct 29 23:07:10.174894 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:10.174857    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7" event={"ID":"3a4d57b5-7fd3-470c-8716-dc3053877b4e","Type":"ContainerStarted","Data":"749ceee12ba636af3c09ddd774303593312ed5e8f915aa8bbb410c248fdcf151"}
Oct 29 23:07:10.175075 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:10.174917    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7" event={"ID":"3a4d57b5-7fd3-470c-8716-dc3053877b4e","Type":"ContainerStarted","Data":"1bc2182a53c3312671d24a9ac9453ca6cd02cdc11469f81972bc01ac3962818c"}
Oct 29 23:07:10.214367 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:10.214314    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7" podStartSLOduration=2.214297652 podStartE2EDuration="2.214297652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:10.212655769 +0000 UTC m=+3169.124099324" watchObservedRunningTime="2026-10-29 23:07:10.214297652 +0000 UTC m=+3169.125741243"
Oct 29 23:07:11.027683 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:11.027640    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body=
Oct 29 23:07:11.027922 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:11.027705    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg" podUID="5716b06a-0198-4647-884c-3251a6a4b63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused"
Oct 29 23:07:11.183592 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:11.183559    4559 generic.go:356] "Generic (PLEG): container finished" podID="3a4d57b5-7fd3-470c-8716-dc3053877b4e" containerID="749ceee12ba636af3c09ddd774303593312ed5e8f915aa8bbb410c248fdcf151" exitCode=0
Oct 29 23:07:11.183592 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:11.183597    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7" event={"ID":"3a4d57b5-7fd3-470c-8716-dc3053877b4e","Type":"ContainerDied","Data":"749ceee12ba636af3c09ddd774303593312ed5e8f915aa8bbb410c248fdcf151"}
Oct 29 23:07:12.028220 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:12.028179    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body=
Oct 29 23:07:12.028660 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:12.028251    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg" podUID="5716b06a-0198-4647-884c-3251a6a4b63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused"
Oct 29 23:07:12.334937 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:12.334904    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"
Oct 29 23:07:12.496558 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:12.496517    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a4d57b5-7fd3-470c-8716-dc3053877b4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a4d57b5-7fd3-470c-8716-dc3053877b4e-maas-api-xks-ca\") pod \"3a4d57b5-7fd3-470c-8716-dc3053877b4e\" (UID: \"3a4d57b5-7fd3-470c-8716-dc3053877b4e\") "
Oct 29 23:07:12.496775 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:12.496569    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a4d57b5-7fd3-470c-8716-dc3053877b4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a4d57b5-7fd3-470c-8716-dc3053877b4e-maas-api-openshift-ca\") pod \"3a4d57b5-7fd3-470c-8716-dc3053877b4e\" (UID: \"3a4d57b5-7fd3-470c-8716-dc3053877b4e\") "
Oct 29 23:07:12.496775 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:12.496623    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a4d57b5-7fd3-470c-8716-dc3053877b4e-kube-api-access-zz5bn\" (UniqueName: \"kubernetes.io/projected/3a4d57b5-7fd3-470c-8716-dc3053877b4e-kube-api-access-zz5bn\") pod \"3a4d57b5-7fd3-470c-8716-dc3053877b4e\" (UID: \"3a4d57b5-7fd3-470c-8716-dc3053877b4e\") "
Oct 29 23:07:12.497048 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:12.497010    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a4d57b5-7fd3-470c-8716-dc3053877b4e-maas-api-openshift-ca" pod "3a4d57b5-7fd3-470c-8716-dc3053877b4e" (UID: "3a4d57b5-7fd3-470c-8716-dc3053877b4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:12.498931 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:12.498907    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a4d57b5-7fd3-470c-8716-dc3053877b4e-maas-api-xks-ca" pod "3a4d57b5-7fd3-470c-8716-dc3053877b4e" (UID: "3a4d57b5-7fd3-470c-8716-dc3053877b4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:07:12.499116 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:12.499089    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4d57b5-7fd3-470c-8716-dc3053877b4e-kube-api-access-zz5bn" pod "3a4d57b5-7fd3-470c-8716-dc3053877b4e" (UID: "3a4d57b5-7fd3-470c-8716-dc3053877b4e"). InnerVolumeSpecName "kube-api-access-zz5bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:12.597629 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:12.597535    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a4d57b5-7fd3-470c-8716-dc3053877b4e-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:07:12.597629 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:12.597569    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a4d57b5-7fd3-470c-8716-dc3053877b4e-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:07:12.597629 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:12.597584    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz5bn\" (UniqueName: \"kubernetes.io/projected/3a4d57b5-7fd3-470c-8716-dc3053877b4e-kube-api-access-zz5bn\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:07:13.029393 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:13.029357    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body=
Oct 29 23:07:13.029836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:13.029410    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg" podUID="5716b06a-0198-4647-884c-3251a6a4b63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused"
Oct 29 23:07:13.201886 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:13.201850    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"
Oct 29 23:07:13.201886 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:13.201862    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7" event={"ID":"3a4d57b5-7fd3-470c-8716-dc3053877b4e","Type":"ContainerDied","Data":"1bc2182a53c3312671d24a9ac9453ca6cd02cdc11469f81972bc01ac3962818c"}
Oct 29 23:07:13.202111 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:13.201918    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bc2182a53c3312671d24a9ac9453ca6cd02cdc11469f81972bc01ac3962818c"
Oct 29 23:07:13.693892 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:07:13.693855    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:07:14.032197 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:14.032118    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:14.032669 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:14.032442    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:14.033322 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:14.033304    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg"
Oct 29 23:07:19.711290 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.711243    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:07:19.713562 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.713535    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"]
Oct 29 23:07:19.714008 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.713994    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3a4d57b5-7fd3-470c-8716-dc3053877b4e" containerName="revoke-keys"
Oct 29 23:07:19.714067 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.714010    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a4d57b5-7fd3-470c-8716-dc3053877b4e" containerName="revoke-keys"
Oct 29 23:07:19.714109 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.714097    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a4d57b5-7fd3-470c-8716-dc3053877b4e" containerName="revoke-keys"
Oct 29 23:07:19.717581 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.717563    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"
Oct 29 23:07:19.732851 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.732794    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:07:19.733069 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.732864    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:07:19.733069 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.732904    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:07:19.733069 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.732946    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:07:19.748655 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.748618    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"]
Oct 29 23:07:19.767360 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.767316    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2nsq\" (UniqueName: \"kubernetes.io/projected/bf46e142-9889-4fd2-b26c-4c14b9819bc0-kube-api-access-j2nsq\") pod \"maas-api-revoke-sub-ec843018-bsj4f\" (UID: \"bf46e142-9889-4fd2-b26c-4c14b9819bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"
Oct 29 23:07:19.767613 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.767508    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf46e142-9889-4fd2-b26c-4c14b9819bc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bsj4f\" (UID: \"bf46e142-9889-4fd2-b26c-4c14b9819bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"
Oct 29 23:07:19.767613 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.767582    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf46e142-9889-4fd2-b26c-4c14b9819bc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bsj4f\" (UID: \"bf46e142-9889-4fd2-b26c-4c14b9819bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"
Oct 29 23:07:19.776577 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.776537    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"]
Oct 29 23:07:19.781141 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.781111    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.799660 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.799630    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"]
Oct 29 23:07:19.868743 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.868707    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2nsq\" (UniqueName: \"kubernetes.io/projected/bf46e142-9889-4fd2-b26c-4c14b9819bc0-kube-api-access-j2nsq\") pod \"maas-api-revoke-sub-ec843018-bsj4f\" (UID: \"bf46e142-9889-4fd2-b26c-4c14b9819bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"
Oct 29 23:07:19.868743 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.868750    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f39456-b754-4dc5-bbca-9486af68dff5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.869019 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.868770    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f39456-b754-4dc5-bbca-9486af68dff5-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.869019 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.868800    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f39456-b754-4dc5-bbca-9486af68dff5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.869019 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.868917    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f39456-b754-4dc5-bbca-9486af68dff5-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.869019 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.868974    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f39456-b754-4dc5-bbca-9486af68dff5-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.869019 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.869007    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5cnr\" (UniqueName: \"kubernetes.io/projected/21f39456-b754-4dc5-bbca-9486af68dff5-kube-api-access-t5cnr\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.869188 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.869059    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f39456-b754-4dc5-bbca-9486af68dff5-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.869188 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.869086    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf46e142-9889-4fd2-b26c-4c14b9819bc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bsj4f\" (UID: \"bf46e142-9889-4fd2-b26c-4c14b9819bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"
Oct 29 23:07:19.869188 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.869114    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f39456-b754-4dc5-bbca-9486af68dff5-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.869188 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.869138    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf46e142-9889-4fd2-b26c-4c14b9819bc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bsj4f\" (UID: \"bf46e142-9889-4fd2-b26c-4c14b9819bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"
Oct 29 23:07:19.869321 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.869189    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f39456-b754-4dc5-bbca-9486af68dff5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.869892 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.869793    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf46e142-9889-4fd2-b26c-4c14b9819bc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bsj4f\" (UID: \"bf46e142-9889-4fd2-b26c-4c14b9819bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"
Oct 29 23:07:19.871789 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.871762    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf46e142-9889-4fd2-b26c-4c14b9819bc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bsj4f\" (UID: \"bf46e142-9889-4fd2-b26c-4c14b9819bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"
Oct 29 23:07:19.888401 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.888370    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2nsq\" (UniqueName: \"kubernetes.io/projected/bf46e142-9889-4fd2-b26c-4c14b9819bc0-kube-api-access-j2nsq\") pod \"maas-api-revoke-sub-ec843018-bsj4f\" (UID: \"bf46e142-9889-4fd2-b26c-4c14b9819bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"
Oct 29 23:07:19.970481 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.970385    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f39456-b754-4dc5-bbca-9486af68dff5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.970481 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.970465    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f39456-b754-4dc5-bbca-9486af68dff5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.970759 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.970492    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f39456-b754-4dc5-bbca-9486af68dff5-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.970759 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.970630    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f39456-b754-4dc5-bbca-9486af68dff5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.970759 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.970685    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f39456-b754-4dc5-bbca-9486af68dff5-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.970759 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.970743    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f39456-b754-4dc5-bbca-9486af68dff5-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.971015 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.970783    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5cnr\" (UniqueName: \"kubernetes.io/projected/21f39456-b754-4dc5-bbca-9486af68dff5-kube-api-access-t5cnr\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.971015 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.970855    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f39456-b754-4dc5-bbca-9486af68dff5-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.971015 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.970904    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f39456-b754-4dc5-bbca-9486af68dff5-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.971015 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.970982    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f39456-b754-4dc5-bbca-9486af68dff5-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.971315 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.971280    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f39456-b754-4dc5-bbca-9486af68dff5-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.971437 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.971343    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f39456-b754-4dc5-bbca-9486af68dff5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.971535 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.971510    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f39456-b754-4dc5-bbca-9486af68dff5-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.971625 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.971272    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f39456-b754-4dc5-bbca-9486af68dff5-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.973159 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.973140    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f39456-b754-4dc5-bbca-9486af68dff5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.973712 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.973693    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f39456-b754-4dc5-bbca-9486af68dff5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.991567 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.991535    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f39456-b754-4dc5-bbca-9486af68dff5-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:19.991759 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:19.991732    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5cnr\" (UniqueName: \"kubernetes.io/projected/21f39456-b754-4dc5-bbca-9486af68dff5-kube-api-access-t5cnr\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tm9cb\" (UID: \"21f39456-b754-4dc5-bbca-9486af68dff5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:20.032387 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:20.032350    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"
Oct 29 23:07:20.101110 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:20.101082    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:20.356790 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:20.356760    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"]
Oct 29 23:07:20.358370 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:07:20.358335    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf46e142_9889_4fd2_b26c_4c14b9819bc0.slice/crio-7a9b3fc65c06d1c5af9cf8e29ad2dc3ff63a84a2df364dc909efc8c6f0817505 WatchSource:0}: Error finding container 7a9b3fc65c06d1c5af9cf8e29ad2dc3ff63a84a2df364dc909efc8c6f0817505: Status 404 returned error can't find the container with id 7a9b3fc65c06d1c5af9cf8e29ad2dc3ff63a84a2df364dc909efc8c6f0817505
Oct 29 23:07:20.432108 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:20.432075    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"]
Oct 29 23:07:20.434697 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:07:20.434657    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f39456_b754_4dc5_bbca_9486af68dff5.slice/crio-60cbe091b1a044d2386d8060c9b99a2842805361906fe5c8e6e4184add285ec0 WatchSource:0}: Error finding container 60cbe091b1a044d2386d8060c9b99a2842805361906fe5c8e6e4184add285ec0: Status 404 returned error can't find the container with id 60cbe091b1a044d2386d8060c9b99a2842805361906fe5c8e6e4184add285ec0
Oct 29 23:07:20.437349 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:20.437308    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:07:20.437448 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:20.437401    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:07:20.437491 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:20.437448    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 29 23:07:21.280337 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:21.272856    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb" event={"ID":"21f39456-b754-4dc5-bbca-9486af68dff5","Type":"ContainerStarted","Data":"d3a53563ac001ff42471ac2fb338b94d53b1e978be38bf4a5aa31c59ab4a4b85"}
Oct 29 23:07:21.280337 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:21.272917    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb" event={"ID":"21f39456-b754-4dc5-bbca-9486af68dff5","Type":"ContainerStarted","Data":"60cbe091b1a044d2386d8060c9b99a2842805361906fe5c8e6e4184add285ec0"}
Oct 29 23:07:21.284641 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:21.283482    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f" event={"ID":"bf46e142-9889-4fd2-b26c-4c14b9819bc0","Type":"ContainerStarted","Data":"a550812727bef3aa2180340ded4dde508c4c26fe22dabb007eeb07abb3c06c0c"}
Oct 29 23:07:21.284641 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:21.283575    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f" event={"ID":"bf46e142-9889-4fd2-b26c-4c14b9819bc0","Type":"ContainerStarted","Data":"7a9b3fc65c06d1c5af9cf8e29ad2dc3ff63a84a2df364dc909efc8c6f0817505"}
Oct 29 23:07:21.316829 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:21.316752    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb" podStartSLOduration=2.316734271 podStartE2EDuration="2.316734271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:21.315040771 +0000 UTC m=+3180.226484348" watchObservedRunningTime="2026-10-29 23:07:21.316734271 +0000 UTC m=+3180.228177827"
Oct 29 23:07:21.361906 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:21.361832    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f" podStartSLOduration=2.361790692 podStartE2EDuration="2.361790692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:21.359235996 +0000 UTC m=+3180.270679552" watchObservedRunningTime="2026-10-29 23:07:21.361790692 +0000 UTC m=+3180.273234248"
Oct 29 23:07:22.101629 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:22.101596    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:22.102950 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:22.102920    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body=
Oct 29 23:07:22.103027 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:22.102980    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb" podUID="21f39456-b754-4dc5-bbca-9486af68dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused"
Oct 29 23:07:22.295746 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:22.295696    4559 generic.go:356] "Generic (PLEG): container finished" podID="bf46e142-9889-4fd2-b26c-4c14b9819bc0" containerID="a550812727bef3aa2180340ded4dde508c4c26fe22dabb007eeb07abb3c06c0c" exitCode=0
Oct 29 23:07:22.296514 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:22.296478    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f" event={"ID":"bf46e142-9889-4fd2-b26c-4c14b9819bc0","Type":"ContainerDied","Data":"a550812727bef3aa2180340ded4dde508c4c26fe22dabb007eeb07abb3c06c0c"}
Oct 29 23:07:23.102459 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:23.102409    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body=
Oct 29 23:07:23.102686 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:23.102476    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb" podUID="21f39456-b754-4dc5-bbca-9486af68dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused"
Oct 29 23:07:23.576878 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:23.576455    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"
Oct 29 23:07:23.623634 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:23.623593    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf46e142-9889-4fd2-b26c-4c14b9819bc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf46e142-9889-4fd2-b26c-4c14b9819bc0-maas-api-xks-ca\") pod \"bf46e142-9889-4fd2-b26c-4c14b9819bc0\" (UID: \"bf46e142-9889-4fd2-b26c-4c14b9819bc0\") "
Oct 29 23:07:23.623862 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:23.623654    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf46e142-9889-4fd2-b26c-4c14b9819bc0-kube-api-access-j2nsq\" (UniqueName: \"kubernetes.io/projected/bf46e142-9889-4fd2-b26c-4c14b9819bc0-kube-api-access-j2nsq\") pod \"bf46e142-9889-4fd2-b26c-4c14b9819bc0\" (UID: \"bf46e142-9889-4fd2-b26c-4c14b9819bc0\") "
Oct 29 23:07:23.623862 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:23.623835    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf46e142-9889-4fd2-b26c-4c14b9819bc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf46e142-9889-4fd2-b26c-4c14b9819bc0-maas-api-openshift-ca\") pod \"bf46e142-9889-4fd2-b26c-4c14b9819bc0\" (UID: \"bf46e142-9889-4fd2-b26c-4c14b9819bc0\") "
Oct 29 23:07:23.624705 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:23.624658    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf46e142-9889-4fd2-b26c-4c14b9819bc0-maas-api-openshift-ca" pod "bf46e142-9889-4fd2-b26c-4c14b9819bc0" (UID: "bf46e142-9889-4fd2-b26c-4c14b9819bc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:23.627788 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:23.627755    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf46e142-9889-4fd2-b26c-4c14b9819bc0-maas-api-xks-ca" pod "bf46e142-9889-4fd2-b26c-4c14b9819bc0" (UID: "bf46e142-9889-4fd2-b26c-4c14b9819bc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:07:23.627958 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:23.627847    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf46e142-9889-4fd2-b26c-4c14b9819bc0-kube-api-access-j2nsq" pod "bf46e142-9889-4fd2-b26c-4c14b9819bc0" (UID: "bf46e142-9889-4fd2-b26c-4c14b9819bc0"). InnerVolumeSpecName "kube-api-access-j2nsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:23.726071 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:23.725921    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf46e142-9889-4fd2-b26c-4c14b9819bc0-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:07:23.726071 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:23.725966    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2nsq\" (UniqueName: \"kubernetes.io/projected/bf46e142-9889-4fd2-b26c-4c14b9819bc0-kube-api-access-j2nsq\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:07:23.726071 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:23.725983    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf46e142-9889-4fd2-b26c-4c14b9819bc0-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:07:24.101577 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:24.101470    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body=
Oct 29 23:07:24.102262 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:24.101847    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb" podUID="21f39456-b754-4dc5-bbca-9486af68dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused"
Oct 29 23:07:24.348996 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:24.347471    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f" event={"ID":"bf46e142-9889-4fd2-b26c-4c14b9819bc0","Type":"ContainerDied","Data":"7a9b3fc65c06d1c5af9cf8e29ad2dc3ff63a84a2df364dc909efc8c6f0817505"}
Oct 29 23:07:24.348996 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:24.347545    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a9b3fc65c06d1c5af9cf8e29ad2dc3ff63a84a2df364dc909efc8c6f0817505"
Oct 29 23:07:24.348996 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:24.347617    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"
Oct 29 23:07:25.045519 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:25.045477    4559 scope.go:122] "RemoveContainer" containerID="8cd2f4cae8de9694673bb093ebaac436a0ebe63bff24d66fd74c4aeecba94353"
Oct 29 23:07:25.103438 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:25.103401    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body=
Oct 29 23:07:25.103776 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:25.103743    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb" podUID="21f39456-b754-4dc5-bbca-9486af68dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused"
Oct 29 23:07:25.130295 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:25.130184    4559 scope.go:122] "RemoveContainer" containerID="edfd79d429447c34b5965f4052369dbf18b364dc1b68db967eaa4852606ae7fc"
Oct 29 23:07:25.206505 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:25.206478    4559 scope.go:122] "RemoveContainer" containerID="0c8cfd7afc1458df33a8c67bad5d92a3ed882aa2c0dee5cf8e25ba2dc4b439d7"
Oct 29 23:07:25.253879 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:25.253850    4559 scope.go:122] "RemoveContainer" containerID="980d3d48d02cbdb6656f0ebb7c6493c10fdc87b7350d63b17b54b4e0b652cb96"
Oct 29 23:07:25.387686 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:25.387340    4559 scope.go:122] "RemoveContainer" containerID="3154d7d58374f93c9ccefd931ae0288d408e75f2dd91da96e71c3315a4dd4998"
Oct 29 23:07:25.504102 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:25.503951    4559 scope.go:122] "RemoveContainer" containerID="31f20ed073933aa4cdbf19be9e3115a48d2dfa505b5b447ab7a8cb131c071542"
Oct 29 23:07:25.565044 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:25.565009    4559 scope.go:122] "RemoveContainer" containerID="ca80472e9f25841cbcdc4c09db4ec0169a31d373e2b3bb6435f17d73105ebf85"
Oct 29 23:07:25.608879 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:25.606959    4559 scope.go:122] "RemoveContainer" containerID="2b5173ea0b5b6c45f93a7f73ca2e6036ead0ac1e23314330aeca408528d653c7"
Oct 29 23:07:26.108177 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:26.105178    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body=
Oct 29 23:07:26.108177 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:26.105242    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb" podUID="21f39456-b754-4dc5-bbca-9486af68dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused"
Oct 29 23:07:27.101484 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:27.101446    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body=
Oct 29 23:07:27.101667 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:27.101531    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb" podUID="21f39456-b754-4dc5-bbca-9486af68dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused"
Oct 29 23:07:28.102332 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:28.102291    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body=
Oct 29 23:07:28.102900 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:28.102354    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb" podUID="21f39456-b754-4dc5-bbca-9486af68dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused"
Oct 29 23:07:30.143059 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:30.143013    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:07:30.143688 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:30.143088    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb" podUID="21f39456-b754-4dc5-bbca-9486af68dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:07:31.184001 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:31.183957    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:07:31.184527 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:31.184019    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb" podUID="21f39456-b754-4dc5-bbca-9486af68dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:07:32.225019 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:32.224977    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:07:32.225469 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:32.225042    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb" podUID="21f39456-b754-4dc5-bbca-9486af68dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:07:33.233213 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:33.233179    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:07:33.233677 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:33.233239    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb" podUID="21f39456-b754-4dc5-bbca-9486af68dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:07:33.236845 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:33.236794    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:33.237107 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:33.237086    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:33.237649 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:33.237629    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tm9cb"
Oct 29 23:07:35.317172 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:35.317120    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:07:43.693005 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:07:43.692966    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:07:47.364704 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.364662    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"]
Oct 29 23:07:47.367163 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.365129    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bf46e142-9889-4fd2-b26c-4c14b9819bc0" containerName="revoke-keys"
Oct 29 23:07:47.367163 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.365142    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf46e142-9889-4fd2-b26c-4c14b9819bc0" containerName="revoke-keys"
Oct 29 23:07:47.367163 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.365257    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf46e142-9889-4fd2-b26c-4c14b9819bc0" containerName="revoke-keys"
Oct 29 23:07:47.368407 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.368389    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"
Oct 29 23:07:47.398396 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.398362    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:07:47.398639 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.398489    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:07:47.398639 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.398534    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:07:47.398913 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.398651    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:07:47.402297 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.402265    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"]
Oct 29 23:07:47.405162 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.405137    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e665a4-a634-4eef-9ea1-ef3666aa47be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-gh4zf\" (UID: \"a8e665a4-a634-4eef-9ea1-ef3666aa47be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"
Oct 29 23:07:47.405278 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.405185    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e665a4-a634-4eef-9ea1-ef3666aa47be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-gh4zf\" (UID: \"a8e665a4-a634-4eef-9ea1-ef3666aa47be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"
Oct 29 23:07:47.405372 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.405349    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwd4\" (UniqueName: \"kubernetes.io/projected/a8e665a4-a634-4eef-9ea1-ef3666aa47be-kube-api-access-xqwd4\") pod \"maas-api-revoke-sub-046e1343-gh4zf\" (UID: \"a8e665a4-a634-4eef-9ea1-ef3666aa47be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"
Oct 29 23:07:47.506326 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.506289    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e665a4-a634-4eef-9ea1-ef3666aa47be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-gh4zf\" (UID: \"a8e665a4-a634-4eef-9ea1-ef3666aa47be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"
Oct 29 23:07:47.506526 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.506337    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e665a4-a634-4eef-9ea1-ef3666aa47be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-gh4zf\" (UID: \"a8e665a4-a634-4eef-9ea1-ef3666aa47be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"
Oct 29 23:07:47.506526 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.506384    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwd4\" (UniqueName: \"kubernetes.io/projected/a8e665a4-a634-4eef-9ea1-ef3666aa47be-kube-api-access-xqwd4\") pod \"maas-api-revoke-sub-046e1343-gh4zf\" (UID: \"a8e665a4-a634-4eef-9ea1-ef3666aa47be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"
Oct 29 23:07:47.507046 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.507025    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e665a4-a634-4eef-9ea1-ef3666aa47be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-gh4zf\" (UID: \"a8e665a4-a634-4eef-9ea1-ef3666aa47be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"
Oct 29 23:07:47.508967 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.508945    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e665a4-a634-4eef-9ea1-ef3666aa47be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-gh4zf\" (UID: \"a8e665a4-a634-4eef-9ea1-ef3666aa47be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"
Oct 29 23:07:47.526780 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.526738    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwd4\" (UniqueName: \"kubernetes.io/projected/a8e665a4-a634-4eef-9ea1-ef3666aa47be-kube-api-access-xqwd4\") pod \"maas-api-revoke-sub-046e1343-gh4zf\" (UID: \"a8e665a4-a634-4eef-9ea1-ef3666aa47be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"
Oct 29 23:07:47.682139 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.682103    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"
Oct 29 23:07:47.980170 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:47.980137    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"]
Oct 29 23:07:47.981761 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:07:47.981735    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e665a4_a634_4eef_9ea1_ef3666aa47be.slice/crio-bcbfe30f912b7616a111db1aeb7826395be1fa7ef8e9d38dbb3951c3a47a7351 WatchSource:0}: Error finding container bcbfe30f912b7616a111db1aeb7826395be1fa7ef8e9d38dbb3951c3a47a7351: Status 404 returned error can't find the container with id bcbfe30f912b7616a111db1aeb7826395be1fa7ef8e9d38dbb3951c3a47a7351
Oct 29 23:07:48.641801 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:48.641525    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf" event={"ID":"a8e665a4-a634-4eef-9ea1-ef3666aa47be","Type":"ContainerStarted","Data":"43dc83bf387ede5ffa52381fe7737322bf731fa825cdb6d152db2291639566b4"}
Oct 29 23:07:48.641801 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:48.641633    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf" event={"ID":"a8e665a4-a634-4eef-9ea1-ef3666aa47be","Type":"ContainerStarted","Data":"bcbfe30f912b7616a111db1aeb7826395be1fa7ef8e9d38dbb3951c3a47a7351"}
Oct 29 23:07:48.682237 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:48.682180    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf" podStartSLOduration=1.6821643750000002 podStartE2EDuration="1.682164375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:48.679716382 +0000 UTC m=+3207.591159950" watchObservedRunningTime="2026-10-29 23:07:48.682164375 +0000 UTC m=+3207.593607929"
Oct 29 23:07:49.649105 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:49.649072    4559 generic.go:356] "Generic (PLEG): container finished" podID="a8e665a4-a634-4eef-9ea1-ef3666aa47be" containerID="43dc83bf387ede5ffa52381fe7737322bf731fa825cdb6d152db2291639566b4" exitCode=0
Oct 29 23:07:49.649535 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:49.649161    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf" event={"ID":"a8e665a4-a634-4eef-9ea1-ef3666aa47be","Type":"ContainerDied","Data":"43dc83bf387ede5ffa52381fe7737322bf731fa825cdb6d152db2291639566b4"}
Oct 29 23:07:50.803857 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:50.803830    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"
Oct 29 23:07:50.841752 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:50.841719    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8e665a4-a634-4eef-9ea1-ef3666aa47be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e665a4-a634-4eef-9ea1-ef3666aa47be-maas-api-xks-ca\") pod \"a8e665a4-a634-4eef-9ea1-ef3666aa47be\" (UID: \"a8e665a4-a634-4eef-9ea1-ef3666aa47be\") "
Oct 29 23:07:50.841985 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:50.841885    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e665a4-a634-4eef-9ea1-ef3666aa47be-kube-api-access-xqwd4\" (UniqueName: \"kubernetes.io/projected/a8e665a4-a634-4eef-9ea1-ef3666aa47be-kube-api-access-xqwd4\") pod \"a8e665a4-a634-4eef-9ea1-ef3666aa47be\" (UID: \"a8e665a4-a634-4eef-9ea1-ef3666aa47be\") "
Oct 29 23:07:50.841985 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:50.841975    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e665a4-a634-4eef-9ea1-ef3666aa47be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e665a4-a634-4eef-9ea1-ef3666aa47be-maas-api-openshift-ca\") pod \"a8e665a4-a634-4eef-9ea1-ef3666aa47be\" (UID: \"a8e665a4-a634-4eef-9ea1-ef3666aa47be\") "
Oct 29 23:07:50.842995 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:50.842955    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e665a4-a634-4eef-9ea1-ef3666aa47be-maas-api-openshift-ca" pod "a8e665a4-a634-4eef-9ea1-ef3666aa47be" (UID: "a8e665a4-a634-4eef-9ea1-ef3666aa47be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:50.845026 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:50.844990    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8e665a4-a634-4eef-9ea1-ef3666aa47be-maas-api-xks-ca" pod "a8e665a4-a634-4eef-9ea1-ef3666aa47be" (UID: "a8e665a4-a634-4eef-9ea1-ef3666aa47be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:07:50.845342 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:50.845314    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e665a4-a634-4eef-9ea1-ef3666aa47be-kube-api-access-xqwd4" pod "a8e665a4-a634-4eef-9ea1-ef3666aa47be" (UID: "a8e665a4-a634-4eef-9ea1-ef3666aa47be"). InnerVolumeSpecName "kube-api-access-xqwd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:50.943695 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:50.943606    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e665a4-a634-4eef-9ea1-ef3666aa47be-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:07:50.943695 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:50.943637    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e665a4-a634-4eef-9ea1-ef3666aa47be-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:07:50.943695 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:50.943646    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqwd4\" (UniqueName: \"kubernetes.io/projected/a8e665a4-a634-4eef-9ea1-ef3666aa47be-kube-api-access-xqwd4\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:07:51.660741 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:51.660708    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"
Oct 29 23:07:51.660982 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:51.660752    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf" event={"ID":"a8e665a4-a634-4eef-9ea1-ef3666aa47be","Type":"ContainerDied","Data":"bcbfe30f912b7616a111db1aeb7826395be1fa7ef8e9d38dbb3951c3a47a7351"}
Oct 29 23:07:51.660982 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:51.660802    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcbfe30f912b7616a111db1aeb7826395be1fa7ef8e9d38dbb3951c3a47a7351"
Oct 29 23:07:57.668697 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.668659    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"]
Oct 29 23:07:57.669152 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.669094    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a8e665a4-a634-4eef-9ea1-ef3666aa47be" containerName="revoke-keys"
Oct 29 23:07:57.669152 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.669106    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e665a4-a634-4eef-9ea1-ef3666aa47be" containerName="revoke-keys"
Oct 29 23:07:57.669241 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.669181    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e665a4-a634-4eef-9ea1-ef3666aa47be" containerName="revoke-keys"
Oct 29 23:07:57.672599 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.672580    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"
Oct 29 23:07:57.689702 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.689670    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:07:57.701965 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.701934    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:07:57.702309 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.702280    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:07:57.702309 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.701977    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:07:57.705485 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.705454    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"]
Oct 29 23:07:57.813060 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.813020    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75539855-d762-4253-bf8c-f12a53e26cbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-z7222\" (UID: \"75539855-d762-4253-bf8c-f12a53e26cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"
Oct 29 23:07:57.813253 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.813073    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rfzx\" (UniqueName: \"kubernetes.io/projected/75539855-d762-4253-bf8c-f12a53e26cbc-kube-api-access-4rfzx\") pod \"maas-api-revoke-sub-d3fba056-z7222\" (UID: \"75539855-d762-4253-bf8c-f12a53e26cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"
Oct 29 23:07:57.813253 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.813096    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75539855-d762-4253-bf8c-f12a53e26cbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-z7222\" (UID: \"75539855-d762-4253-bf8c-f12a53e26cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"
Oct 29 23:07:57.913968 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.913926    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75539855-d762-4253-bf8c-f12a53e26cbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-z7222\" (UID: \"75539855-d762-4253-bf8c-f12a53e26cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"
Oct 29 23:07:57.914169 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.913979    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rfzx\" (UniqueName: \"kubernetes.io/projected/75539855-d762-4253-bf8c-f12a53e26cbc-kube-api-access-4rfzx\") pod \"maas-api-revoke-sub-d3fba056-z7222\" (UID: \"75539855-d762-4253-bf8c-f12a53e26cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"
Oct 29 23:07:57.914169 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.914005    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75539855-d762-4253-bf8c-f12a53e26cbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-z7222\" (UID: \"75539855-d762-4253-bf8c-f12a53e26cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"
Oct 29 23:07:57.914623 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.914591    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75539855-d762-4253-bf8c-f12a53e26cbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-z7222\" (UID: \"75539855-d762-4253-bf8c-f12a53e26cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"
Oct 29 23:07:57.916607 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.916586    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75539855-d762-4253-bf8c-f12a53e26cbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-z7222\" (UID: \"75539855-d762-4253-bf8c-f12a53e26cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"
Oct 29 23:07:57.934276 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.934195    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rfzx\" (UniqueName: \"kubernetes.io/projected/75539855-d762-4253-bf8c-f12a53e26cbc-kube-api-access-4rfzx\") pod \"maas-api-revoke-sub-d3fba056-z7222\" (UID: \"75539855-d762-4253-bf8c-f12a53e26cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"
Oct 29 23:07:57.986223 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:57.986193    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"
Oct 29 23:07:58.288672 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:58.288644    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"]
Oct 29 23:07:58.291614 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:07:58.291579    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75539855_d762_4253_bf8c_f12a53e26cbc.slice/crio-b76ec8681fd5e8ad51f570a9540c707c8293c8c3d9d1cf3391b20b0cd07ee4f1 WatchSource:0}: Error finding container b76ec8681fd5e8ad51f570a9540c707c8293c8c3d9d1cf3391b20b0cd07ee4f1: Status 404 returned error can't find the container with id b76ec8681fd5e8ad51f570a9540c707c8293c8c3d9d1cf3391b20b0cd07ee4f1
Oct 29 23:07:58.586133 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:58.586092    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:07:58.703637 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:58.703595    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222" event={"ID":"75539855-d762-4253-bf8c-f12a53e26cbc","Type":"ContainerStarted","Data":"42996a27f545acab001a3a0529201e5560c807a7b0510ef1766596c472ba583a"}
Oct 29 23:07:58.704098 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:58.703659    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222" event={"ID":"75539855-d762-4253-bf8c-f12a53e26cbc","Type":"ContainerStarted","Data":"b76ec8681fd5e8ad51f570a9540c707c8293c8c3d9d1cf3391b20b0cd07ee4f1"}
Oct 29 23:07:58.754337 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:58.754277    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222" podStartSLOduration=1.754260021 podStartE2EDuration="1.754260021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:58.740828817 +0000 UTC m=+3217.652272366" watchObservedRunningTime="2026-10-29 23:07:58.754260021 +0000 UTC m=+3217.665703601"
Oct 29 23:07:59.716804 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:59.716753    4559 generic.go:356] "Generic (PLEG): container finished" podID="75539855-d762-4253-bf8c-f12a53e26cbc" containerID="42996a27f545acab001a3a0529201e5560c807a7b0510ef1766596c472ba583a" exitCode=0
Oct 29 23:07:59.717381 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:07:59.716854    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222" event={"ID":"75539855-d762-4253-bf8c-f12a53e26cbc","Type":"ContainerDied","Data":"42996a27f545acab001a3a0529201e5560c807a7b0510ef1766596c472ba583a"}
Oct 29 23:08:01.628860 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:01.628179    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"
Oct 29 23:08:01.669134 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:01.668163    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75539855-d762-4253-bf8c-f12a53e26cbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75539855-d762-4253-bf8c-f12a53e26cbc-maas-api-openshift-ca\") pod \"75539855-d762-4253-bf8c-f12a53e26cbc\" (UID: \"75539855-d762-4253-bf8c-f12a53e26cbc\") "
Oct 29 23:08:01.669134 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:01.668305    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75539855-d762-4253-bf8c-f12a53e26cbc-kube-api-access-4rfzx\" (UniqueName: \"kubernetes.io/projected/75539855-d762-4253-bf8c-f12a53e26cbc-kube-api-access-4rfzx\") pod \"75539855-d762-4253-bf8c-f12a53e26cbc\" (UID: \"75539855-d762-4253-bf8c-f12a53e26cbc\") "
Oct 29 23:08:01.669134 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:01.668346    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75539855-d762-4253-bf8c-f12a53e26cbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75539855-d762-4253-bf8c-f12a53e26cbc-maas-api-xks-ca\") pod \"75539855-d762-4253-bf8c-f12a53e26cbc\" (UID: \"75539855-d762-4253-bf8c-f12a53e26cbc\") "
Oct 29 23:08:01.669912 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:01.669884    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75539855-d762-4253-bf8c-f12a53e26cbc-maas-api-openshift-ca" pod "75539855-d762-4253-bf8c-f12a53e26cbc" (UID: "75539855-d762-4253-bf8c-f12a53e26cbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:01.708954 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:01.708557    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75539855-d762-4253-bf8c-f12a53e26cbc-maas-api-xks-ca" pod "75539855-d762-4253-bf8c-f12a53e26cbc" (UID: "75539855-d762-4253-bf8c-f12a53e26cbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:01.764872 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:01.764828    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75539855-d762-4253-bf8c-f12a53e26cbc-kube-api-access-4rfzx" pod "75539855-d762-4253-bf8c-f12a53e26cbc" (UID: "75539855-d762-4253-bf8c-f12a53e26cbc"). InnerVolumeSpecName "kube-api-access-4rfzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:01.767847 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:01.767784    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222" event={"ID":"75539855-d762-4253-bf8c-f12a53e26cbc","Type":"ContainerDied","Data":"b76ec8681fd5e8ad51f570a9540c707c8293c8c3d9d1cf3391b20b0cd07ee4f1"}
Oct 29 23:08:01.768150 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:01.768112    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b76ec8681fd5e8ad51f570a9540c707c8293c8c3d9d1cf3391b20b0cd07ee4f1"
Oct 29 23:08:01.768397 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:01.768380    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"
Oct 29 23:08:01.770890 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:01.770638    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75539855-d762-4253-bf8c-f12a53e26cbc-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:08:01.770890 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:01.770666    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rfzx\" (UniqueName: \"kubernetes.io/projected/75539855-d762-4253-bf8c-f12a53e26cbc-kube-api-access-4rfzx\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:08:01.770890 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:01.770680    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75539855-d762-4253-bf8c-f12a53e26cbc-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:08:08.893016 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:08.892974    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:08:13.181558 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:13.181514    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:08:13.698762 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:08:13.698719    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:08:25.859279 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:25.859239    4559 scope.go:122] "RemoveContainer" containerID="79941e7d25cfa124822e889ae9960b5dd22b2558af929d5cd8c3249428a88b73"
Oct 29 23:08:25.868458 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:25.868435    4559 scope.go:122] "RemoveContainer" containerID="3bfe397013fbf882777a3302c6ddab0d77e5ebe4a45e02f7025ece4402cdb4e7"
Oct 29 23:08:25.877545 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:25.877517    4559 scope.go:122] "RemoveContainer" containerID="4003f9aa213aaf8157da23fe6b9f704db63742c3f7bb9b44674e23ff183496b5"
Oct 29 23:08:25.886400 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:25.886377    4559 scope.go:122] "RemoveContainer" containerID="ca26215ac89336825eb6cb8bc185a563a6426909b886d5d0acc47a3d4ab4b696"
Oct 29 23:08:25.894603 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:25.894583    4559 scope.go:122] "RemoveContainer" containerID="a8153c76316ac00a2e6385e72e308c43037baa0b388e94898e365ae83c183d09"
Oct 29 23:08:30.365706 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:30.365668    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:08:31.028412 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:31.028374    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"]
Oct 29 23:08:31.041292 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:31.041253    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c884a500-w878q"]
Oct 29 23:08:31.227219 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:31.227186    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f81c8641-eaee-4e4e-8af5-a732e9cb6fcd" path="/var/lib/kubelet/pods/f81c8641-eaee-4e4e-8af5-a732e9cb6fcd/volumes"
Oct 29 23:08:32.027204 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:32.027157    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"]
Oct 29 23:08:32.030061 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:32.030033    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b220fd4-lmxtw"]
Oct 29 23:08:33.254718 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:33.254675    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="386d9794-ca90-400c-87f9-6a2cb7546982" path="/var/lib/kubelet/pods/386d9794-ca90-400c-87f9-6a2cb7546982/volumes"
Oct 29 23:08:35.421879 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:35.417961    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:08:43.692714 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:08:43.692670    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:08:51.807351 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:51.807311    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:08:56.001079 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:56.001038    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:08:59.494982 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:59.494931    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:08:59.495650 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:08:59.495613    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g" podUID="e0384c7e-3969-4a0e-8e4d-7d9ffa718730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:09:12.810632 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:09:12.810541    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:09:13.708071 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:09:13.707998    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:09:15.158597 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:09:15.116537    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"]
Oct 29 23:09:15.178144 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:09:15.178090    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"]
Oct 29 23:09:15.226948 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:09:15.226890    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efaeed0e-bm78q"]
Oct 29 23:09:15.359839 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:09:15.358106    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05b244d1-56ea-4f24-a7e0-527308b9aa4a" path="/var/lib/kubelet/pods/05b244d1-56ea-4f24-a7e0-527308b9aa4a/volumes"
Oct 29 23:09:15.382836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:09:15.381056    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03358a9c-m8xmk"]
Oct 29 23:09:17.227114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:09:17.227075    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="541dda73-1e06-48f8-84c5-6c295e16286b" path="/var/lib/kubelet/pods/541dda73-1e06-48f8-84c5-6c295e16286b/volumes"
Oct 29 23:09:21.947441 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:09:21.947415    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 23:09:21.947946 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:09:21.947791    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 23:09:25.930836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:09:25.930766    4559 scope.go:122] "RemoveContainer" containerID="c8261afece87aeb734717c2eea8ab1651392136ce678d6cba4337e3a590466bf"
Oct 29 23:09:25.940303 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:09:25.940276    4559 scope.go:122] "RemoveContainer" containerID="77f6b76af1bfc6362c702ed13ead796df108398b48b4a9417aa9ed14377d1787"
Oct 29 23:09:28.831692 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:09:28.831652    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:09:39.486142 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:09:39.486102    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:09:43.695699 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:09:43.695653    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:09:44.713736 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:09:44.713696    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:10:01.508417 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:01.508379    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:10:06.268549 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:06.268511    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:10:12.035504 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:12.035464    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"]
Oct 29 23:10:12.048120 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:12.048086    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984d3085-m4bcv"]
Oct 29 23:10:13.225977 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:13.225943    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c06d48ce-2877-4075-8379-16218f4d51d8" path="/var/lib/kubelet/pods/c06d48ce-2877-4075-8379-16218f4d51d8/volumes"
Oct 29 23:10:13.693412 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:10:13.693378    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:10:25.967251 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:25.967215    4559 scope.go:122] "RemoveContainer" containerID="b8c97c6100c84e43105c3b84e4f0d97229e91c5059ee4429d4123eb503a9f020"
Oct 29 23:10:25.976685 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:25.976661    4559 scope.go:122] "RemoveContainer" containerID="fd08010233fbf2622e548c1fd977e4fc9f70738656f44a3394a69da865856fac"
Oct 29 23:10:25.985401 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:25.985374    4559 scope.go:122] "RemoveContainer" containerID="eea2d0617c92e318c5e97485f18930b9c96631a2534e8e71ea3897fe405eab1b"
Oct 29 23:10:25.994001 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:25.993976    4559 scope.go:122] "RemoveContainer" containerID="336135d851a763bed51e21d42919b277e32b50275e12189bf0a1718928b96c2d"
Oct 29 23:10:26.002564 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:26.002537    4559 scope.go:122] "RemoveContainer" containerID="791a8c80fedcda3c64072435609b47000df9dfff328bc2ae2e7d0b8e3f14cd47"
Oct 29 23:10:26.011337 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:26.011317    4559 scope.go:122] "RemoveContainer" containerID="d72524d63a238374e9c9664bb7337cc7314673978034dfb296279258e13ff548"
Oct 29 23:10:33.551974 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.551888    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"]
Oct 29 23:10:33.552456 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.552433    4559 cpu_manager.go:414] "containerMap: removing container" podUID="75539855-d762-4253-bf8c-f12a53e26cbc" containerName="revoke-keys"
Oct 29 23:10:33.552545 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.552457    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75539855-d762-4253-bf8c-f12a53e26cbc" containerName="revoke-keys"
Oct 29 23:10:33.553221 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.553192    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="75539855-d762-4253-bf8c-f12a53e26cbc" containerName="revoke-keys"
Oct 29 23:10:33.559930 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.559885    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:10:33.577748 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.577714    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"]
Oct 29 23:10:33.587610 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.587576    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 23:10:33.587610 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.587601    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:10:33.587932 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.587768    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dzwhk\""
Oct 29 23:10:33.588899 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.588707    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:10:33.588899 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.588872    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-e5236b-serving-cert\""
Oct 29 23:10:33.631865 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.631824    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/375baf55-7ebf-4b06-800c-bea573bb31de-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85\" (UID: \"375baf55-7ebf-4b06-800c-bea573bb31de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:10:33.632085 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.631939    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hxjk\" (UniqueName: \"kubernetes.io/projected/375baf55-7ebf-4b06-800c-bea573bb31de-kube-api-access-2hxjk\") pod \"maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85\" (UID: \"375baf55-7ebf-4b06-800c-bea573bb31de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:10:33.632085 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.632025    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375baf55-7ebf-4b06-800c-bea573bb31de-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85\" (UID: \"375baf55-7ebf-4b06-800c-bea573bb31de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:10:33.632085 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.632059    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375baf55-7ebf-4b06-800c-bea573bb31de-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85\" (UID: \"375baf55-7ebf-4b06-800c-bea573bb31de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:10:33.736157 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.736109    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/375baf55-7ebf-4b06-800c-bea573bb31de-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85\" (UID: \"375baf55-7ebf-4b06-800c-bea573bb31de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:10:33.736633 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.736582    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hxjk\" (UniqueName: \"kubernetes.io/projected/375baf55-7ebf-4b06-800c-bea573bb31de-kube-api-access-2hxjk\") pod \"maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85\" (UID: \"375baf55-7ebf-4b06-800c-bea573bb31de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:10:33.736866 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.736847    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375baf55-7ebf-4b06-800c-bea573bb31de-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85\" (UID: \"375baf55-7ebf-4b06-800c-bea573bb31de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:10:33.737010 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.736995    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375baf55-7ebf-4b06-800c-bea573bb31de-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85\" (UID: \"375baf55-7ebf-4b06-800c-bea573bb31de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:10:33.737305 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:10:33.737285    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-serving-cert: secret "maas-api-e2e-worker-single-main-e5236b-serving-cert" not found
Oct 29 23:10:33.737506 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:10:33.737466    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375baf55-7ebf-4b06-800c-bea573bb31de-maas-api-tls podName:375baf55-7ebf-4b06-800c-bea573bb31de nodeName:}" failed. No retries permitted until 2026-10-29 23:10:34.237441052 +0000 UTC m=+3373.148884609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/375baf55-7ebf-4b06-800c-bea573bb31de-maas-api-tls") pod "maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85" (UID: "375baf55-7ebf-4b06-800c-bea573bb31de") : secret "maas-api-e2e-worker-single-main-e5236b-serving-cert" not found
Oct 29 23:10:33.738797 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.738773    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/375baf55-7ebf-4b06-800c-bea573bb31de-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85\" (UID: \"375baf55-7ebf-4b06-800c-bea573bb31de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:10:33.751596 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.746776    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375baf55-7ebf-4b06-800c-bea573bb31de-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85\" (UID: \"375baf55-7ebf-4b06-800c-bea573bb31de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:10:33.762312 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:33.762276    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hxjk\" (UniqueName: \"kubernetes.io/projected/375baf55-7ebf-4b06-800c-bea573bb31de-kube-api-access-2hxjk\") pod \"maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85\" (UID: \"375baf55-7ebf-4b06-800c-bea573bb31de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:10:34.243053 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:34.243010    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375baf55-7ebf-4b06-800c-bea573bb31de-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85\" (UID: \"375baf55-7ebf-4b06-800c-bea573bb31de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:10:34.245867 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:34.245831    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375baf55-7ebf-4b06-800c-bea573bb31de-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85\" (UID: \"375baf55-7ebf-4b06-800c-bea573bb31de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:10:34.485448 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:34.485408    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:10:34.783690 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:34.783657    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"]
Oct 29 23:10:34.786042 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:10:34.786012    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375baf55_7ebf_4b06_800c_bea573bb31de.slice/crio-6e4338fdda877b8c75a19b2a3be868033c1f2f66c6946a0d64ce7d5d972663be WatchSource:0}: Error finding container 6e4338fdda877b8c75a19b2a3be868033c1f2f66c6946a0d64ce7d5d972663be: Status 404 returned error can't find the container with id 6e4338fdda877b8c75a19b2a3be868033c1f2f66c6946a0d64ce7d5d972663be
Oct 29 23:10:34.787393 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:34.787376    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:10:34.936864 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:34.936823    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85" event={"ID":"375baf55-7ebf-4b06-800c-bea573bb31de","Type":"ContainerStarted","Data":"6e4338fdda877b8c75a19b2a3be868033c1f2f66c6946a0d64ce7d5d972663be"}
Oct 29 23:10:35.944551 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:35.944510    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85" event={"ID":"375baf55-7ebf-4b06-800c-bea573bb31de","Type":"ContainerStarted","Data":"5f60a12ca87d2e503ebe6e16ca192aecc31dda43e10dfbb2f1055ac463f851fa"}
Oct 29 23:10:35.945013 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:35.944683    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:10:35.983978 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:35.983920    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85" podStartSLOduration=2.510988417 podStartE2EDuration="2.983903037s" totalImagesPullingTime="472.91462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:35.983353304 +0000 UTC m=+3374.894796858" watchObservedRunningTime="2026-10-29 23:10:35.983903037 +0000 UTC m=+3374.895346592"
Oct 29 23:10:37.094748 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:37.094715    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:10:41.955588 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:41.955554    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:10:43.693271 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:10:43.693231    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:10:52.618286 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:10:52.618249    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:11:02.027538 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:02.027486    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"]
Oct 29 23:11:02.036507 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:02.036474    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jvhs"]
Oct 29 23:11:03.226209 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:03.226174    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f40ea5c-4c3b-4392-be6a-2c8650593ae9" path="/var/lib/kubelet/pods/7f40ea5c-4c3b-4392-be6a-2c8650593ae9/volumes"
Oct 29 23:11:06.781973 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:06.781932    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:11:13.693785 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:11:13.693746    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:11:24.272960 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:24.272913    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:11:26.057395 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:26.057361    4559 scope.go:122] "RemoveContainer" containerID="c5f9c3e3ee4e2df0adfd4c51ff6350e79a939746a16e3aad7fe18c9f4b2ed62f"
Oct 29 23:11:26.066471 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:26.066441    4559 scope.go:122] "RemoveContainer" containerID="adb4b19c8cf09a765355d6b735588fb2597b711b26fae8e1cfd3f63f2e0798bb"
Oct 29 23:11:26.078273 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:26.078247    4559 scope.go:122] "RemoveContainer" containerID="51e3a2d5c7b3ab89d362b5110b66f386b47516d2a0588dbc893206e48983a812"
Oct 29 23:11:35.017623 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:35.017590    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"]
Oct 29 23:11:35.018260 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:35.017843    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85" podUID="375baf55-7ebf-4b06-800c-bea573bb31de" containerName="maas-api" containerID="cri-o://5f60a12ca87d2e503ebe6e16ca192aecc31dda43e10dfbb2f1055ac463f851fa" gracePeriod=30
Oct 29 23:11:35.537591 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:35.537553    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:11:35.713236 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:35.713196    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375baf55-7ebf-4b06-800c-bea573bb31de-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/375baf55-7ebf-4b06-800c-bea573bb31de-openshift-service-ca\") pod \"375baf55-7ebf-4b06-800c-bea573bb31de\" (UID: \"375baf55-7ebf-4b06-800c-bea573bb31de\") "
Oct 29 23:11:35.713236 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:35.713244    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375baf55-7ebf-4b06-800c-bea573bb31de-metrics-certs\" (UniqueName: \"kubernetes.io/secret/375baf55-7ebf-4b06-800c-bea573bb31de-metrics-certs\") pod \"375baf55-7ebf-4b06-800c-bea573bb31de\" (UID: \"375baf55-7ebf-4b06-800c-bea573bb31de\") "
Oct 29 23:11:35.713481 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:35.713281    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375baf55-7ebf-4b06-800c-bea573bb31de-kube-api-access-2hxjk\" (UniqueName: \"kubernetes.io/projected/375baf55-7ebf-4b06-800c-bea573bb31de-kube-api-access-2hxjk\") pod \"375baf55-7ebf-4b06-800c-bea573bb31de\" (UID: \"375baf55-7ebf-4b06-800c-bea573bb31de\") "
Oct 29 23:11:35.713481 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:35.713360    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375baf55-7ebf-4b06-800c-bea573bb31de-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375baf55-7ebf-4b06-800c-bea573bb31de-maas-api-tls\") pod \"375baf55-7ebf-4b06-800c-bea573bb31de\" (UID: \"375baf55-7ebf-4b06-800c-bea573bb31de\") "
Oct 29 23:11:35.713694 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:35.713662    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375baf55-7ebf-4b06-800c-bea573bb31de-openshift-service-ca" pod "375baf55-7ebf-4b06-800c-bea573bb31de" (UID: "375baf55-7ebf-4b06-800c-bea573bb31de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:35.715625 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:35.715593    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375baf55-7ebf-4b06-800c-bea573bb31de-metrics-certs" pod "375baf55-7ebf-4b06-800c-bea573bb31de" (UID: "375baf55-7ebf-4b06-800c-bea573bb31de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:11:35.715625 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:35.715610    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375baf55-7ebf-4b06-800c-bea573bb31de-maas-api-tls" pod "375baf55-7ebf-4b06-800c-bea573bb31de" (UID: "375baf55-7ebf-4b06-800c-bea573bb31de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:11:35.715796 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:35.715660    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375baf55-7ebf-4b06-800c-bea573bb31de-kube-api-access-2hxjk" pod "375baf55-7ebf-4b06-800c-bea573bb31de" (UID: "375baf55-7ebf-4b06-800c-bea573bb31de"). InnerVolumeSpecName "kube-api-access-2hxjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:35.814323 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:35.814286    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375baf55-7ebf-4b06-800c-bea573bb31de-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:11:35.814323 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:35.814317    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/375baf55-7ebf-4b06-800c-bea573bb31de-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:11:35.814323 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:35.814328    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375baf55-7ebf-4b06-800c-bea573bb31de-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:11:35.814583 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:35.814337    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hxjk\" (UniqueName: \"kubernetes.io/projected/375baf55-7ebf-4b06-800c-bea573bb31de-kube-api-access-2hxjk\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:11:36.295657 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:36.295618    4559 generic.go:356] "Generic (PLEG): container finished" podID="375baf55-7ebf-4b06-800c-bea573bb31de" containerID="5f60a12ca87d2e503ebe6e16ca192aecc31dda43e10dfbb2f1055ac463f851fa" exitCode=0
Oct 29 23:11:36.296147 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:36.295669    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"
Oct 29 23:11:36.296147 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:36.295690    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85" event={"ID":"375baf55-7ebf-4b06-800c-bea573bb31de","Type":"ContainerDied","Data":"5f60a12ca87d2e503ebe6e16ca192aecc31dda43e10dfbb2f1055ac463f851fa"}
Oct 29 23:11:36.296147 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:36.295754    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85" event={"ID":"375baf55-7ebf-4b06-800c-bea573bb31de","Type":"ContainerDied","Data":"6e4338fdda877b8c75a19b2a3be868033c1f2f66c6946a0d64ce7d5d972663be"}
Oct 29 23:11:36.296147 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:36.295782    4559 scope.go:122] "RemoveContainer" containerID="5f60a12ca87d2e503ebe6e16ca192aecc31dda43e10dfbb2f1055ac463f851fa"
Oct 29 23:11:36.306725 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:36.306703    4559 scope.go:122] "RemoveContainer" containerID="5f60a12ca87d2e503ebe6e16ca192aecc31dda43e10dfbb2f1055ac463f851fa"
Oct 29 23:11:36.307187 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:11:36.307162    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f60a12ca87d2e503ebe6e16ca192aecc31dda43e10dfbb2f1055ac463f851fa\": container with ID starting with 5f60a12ca87d2e503ebe6e16ca192aecc31dda43e10dfbb2f1055ac463f851fa not found: ID does not exist" containerID="5f60a12ca87d2e503ebe6e16ca192aecc31dda43e10dfbb2f1055ac463f851fa"
Oct 29 23:11:36.307269 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:36.307197    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f60a12ca87d2e503ebe6e16ca192aecc31dda43e10dfbb2f1055ac463f851fa"} err="failed to get container status \"5f60a12ca87d2e503ebe6e16ca192aecc31dda43e10dfbb2f1055ac463f851fa\": rpc error: code = NotFound desc = could not find container \"5f60a12ca87d2e503ebe6e16ca192aecc31dda43e10dfbb2f1055ac463f851fa\": container with ID starting with 5f60a12ca87d2e503ebe6e16ca192aecc31dda43e10dfbb2f1055ac463f851fa not found: ID does not exist"
Oct 29 23:11:36.347217 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:36.347178    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"]
Oct 29 23:11:36.359785 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:36.359754    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e5236b-7bfc49c756-57v85"]
Oct 29 23:11:37.225731 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:37.225693    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="375baf55-7ebf-4b06-800c-bea573bb31de" path="/var/lib/kubelet/pods/375baf55-7ebf-4b06-800c-bea573bb31de/volumes"
Oct 29 23:11:37.490554 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:37.490463    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:11:43.694539 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:11:43.694502    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:11:45.734175 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.734134    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"]
Oct 29 23:11:45.734607 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.734586    4559 cpu_manager.go:414] "containerMap: removing container" podUID="375baf55-7ebf-4b06-800c-bea573bb31de" containerName="maas-api"
Oct 29 23:11:45.734607 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.734597    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375baf55-7ebf-4b06-800c-bea573bb31de" containerName="maas-api"
Oct 29 23:11:45.734694 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.734664    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="375baf55-7ebf-4b06-800c-bea573bb31de" containerName="maas-api"
Oct 29 23:11:45.739290 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.739265    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"
Oct 29 23:11:45.758673 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.758639    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"]
Oct 29 23:11:45.768850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.768792    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:11:45.768850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.768843    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:11:45.768850 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.768824    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:11:45.769105 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.768889    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:11:45.820703 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.820660    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28tkj\" (UniqueName: \"kubernetes.io/projected/ef7d2cb2-e200-4c5b-82be-e778409fa251-kube-api-access-28tkj\") pod \"maas-api-revoke-sub-3d4156f1-7p87z\" (UID: \"ef7d2cb2-e200-4c5b-82be-e778409fa251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"
Oct 29 23:11:45.820703 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.820706    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7d2cb2-e200-4c5b-82be-e778409fa251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7p87z\" (UID: \"ef7d2cb2-e200-4c5b-82be-e778409fa251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"
Oct 29 23:11:45.820986 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.820804    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7d2cb2-e200-4c5b-82be-e778409fa251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7p87z\" (UID: \"ef7d2cb2-e200-4c5b-82be-e778409fa251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"
Oct 29 23:11:45.921662 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.921625    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28tkj\" (UniqueName: \"kubernetes.io/projected/ef7d2cb2-e200-4c5b-82be-e778409fa251-kube-api-access-28tkj\") pod \"maas-api-revoke-sub-3d4156f1-7p87z\" (UID: \"ef7d2cb2-e200-4c5b-82be-e778409fa251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"
Oct 29 23:11:45.921662 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.921666    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7d2cb2-e200-4c5b-82be-e778409fa251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7p87z\" (UID: \"ef7d2cb2-e200-4c5b-82be-e778409fa251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"
Oct 29 23:11:45.922009 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.921698    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7d2cb2-e200-4c5b-82be-e778409fa251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7p87z\" (UID: \"ef7d2cb2-e200-4c5b-82be-e778409fa251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"
Oct 29 23:11:45.922343 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.922320    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7d2cb2-e200-4c5b-82be-e778409fa251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7p87z\" (UID: \"ef7d2cb2-e200-4c5b-82be-e778409fa251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"
Oct 29 23:11:45.924403 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.924379    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7d2cb2-e200-4c5b-82be-e778409fa251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7p87z\" (UID: \"ef7d2cb2-e200-4c5b-82be-e778409fa251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"
Oct 29 23:11:45.942388 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:45.942353    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28tkj\" (UniqueName: \"kubernetes.io/projected/ef7d2cb2-e200-4c5b-82be-e778409fa251-kube-api-access-28tkj\") pod \"maas-api-revoke-sub-3d4156f1-7p87z\" (UID: \"ef7d2cb2-e200-4c5b-82be-e778409fa251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"
Oct 29 23:11:46.053225 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:46.053131    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"
Oct 29 23:11:46.353402 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:46.353362    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"]
Oct 29 23:11:46.356441 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:11:46.356407    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7d2cb2_e200_4c5b_82be_e778409fa251.slice/crio-0f9497ae9b2ff7afddca2ba12eec68f58726e53fba0e48f81548d3b8e5007fec WatchSource:0}: Error finding container 0f9497ae9b2ff7afddca2ba12eec68f58726e53fba0e48f81548d3b8e5007fec: Status 404 returned error can't find the container with id 0f9497ae9b2ff7afddca2ba12eec68f58726e53fba0e48f81548d3b8e5007fec
Oct 29 23:11:47.355611 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:47.355575    4559 generic.go:356] "Generic (PLEG): container finished" podID="ef7d2cb2-e200-4c5b-82be-e778409fa251" containerID="cfe9742e10f3e243a965a27a029cb3a9abaacde226f5e71ec8abb54aa3c7c94e" exitCode=0
Oct 29 23:11:47.356045 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:47.355667    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z" event={"ID":"ef7d2cb2-e200-4c5b-82be-e778409fa251","Type":"ContainerDied","Data":"cfe9742e10f3e243a965a27a029cb3a9abaacde226f5e71ec8abb54aa3c7c94e"}
Oct 29 23:11:47.356045 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:47.355725    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z" event={"ID":"ef7d2cb2-e200-4c5b-82be-e778409fa251","Type":"ContainerStarted","Data":"0f9497ae9b2ff7afddca2ba12eec68f58726e53fba0e48f81548d3b8e5007fec"}
Oct 29 23:11:48.501457 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:48.501420    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"
Oct 29 23:11:48.546391 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:48.546355    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7d2cb2-e200-4c5b-82be-e778409fa251-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7d2cb2-e200-4c5b-82be-e778409fa251-maas-api-xks-ca\") pod \"ef7d2cb2-e200-4c5b-82be-e778409fa251\" (UID: \"ef7d2cb2-e200-4c5b-82be-e778409fa251\") "
Oct 29 23:11:48.546598 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:48.546455    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7d2cb2-e200-4c5b-82be-e778409fa251-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7d2cb2-e200-4c5b-82be-e778409fa251-maas-api-openshift-ca\") pod \"ef7d2cb2-e200-4c5b-82be-e778409fa251\" (UID: \"ef7d2cb2-e200-4c5b-82be-e778409fa251\") "
Oct 29 23:11:48.546598 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:48.546529    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7d2cb2-e200-4c5b-82be-e778409fa251-kube-api-access-28tkj\" (UniqueName: \"kubernetes.io/projected/ef7d2cb2-e200-4c5b-82be-e778409fa251-kube-api-access-28tkj\") pod \"ef7d2cb2-e200-4c5b-82be-e778409fa251\" (UID: \"ef7d2cb2-e200-4c5b-82be-e778409fa251\") "
Oct 29 23:11:48.546911 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:48.546881    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7d2cb2-e200-4c5b-82be-e778409fa251-maas-api-openshift-ca" pod "ef7d2cb2-e200-4c5b-82be-e778409fa251" (UID: "ef7d2cb2-e200-4c5b-82be-e778409fa251"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:48.548744 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:48.548707    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7d2cb2-e200-4c5b-82be-e778409fa251-maas-api-xks-ca" pod "ef7d2cb2-e200-4c5b-82be-e778409fa251" (UID: "ef7d2cb2-e200-4c5b-82be-e778409fa251"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:11:48.549050 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:48.549023    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7d2cb2-e200-4c5b-82be-e778409fa251-kube-api-access-28tkj" pod "ef7d2cb2-e200-4c5b-82be-e778409fa251" (UID: "ef7d2cb2-e200-4c5b-82be-e778409fa251"). InnerVolumeSpecName "kube-api-access-28tkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:48.647968 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:48.647938    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28tkj\" (UniqueName: \"kubernetes.io/projected/ef7d2cb2-e200-4c5b-82be-e778409fa251-kube-api-access-28tkj\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:11:48.647968 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:48.647970    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7d2cb2-e200-4c5b-82be-e778409fa251-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:11:48.648178 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:48.647979    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7d2cb2-e200-4c5b-82be-e778409fa251-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:11:49.368784 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:49.368751    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"
Oct 29 23:11:49.368982 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:49.368800    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z" event={"ID":"ef7d2cb2-e200-4c5b-82be-e778409fa251","Type":"ContainerDied","Data":"0f9497ae9b2ff7afddca2ba12eec68f58726e53fba0e48f81548d3b8e5007fec"}
Oct 29 23:11:49.368982 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:49.368872    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f9497ae9b2ff7afddca2ba12eec68f58726e53fba0e48f81548d3b8e5007fec"
Oct 29 23:11:51.034177 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:51.034131    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"]
Oct 29 23:11:51.044163 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:51.044131    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lc9ct"]
Oct 29 23:11:51.226876 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:11:51.226831    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4" path="/var/lib/kubelet/pods/cbcf5042-3ee9-4ffd-bad7-cfe83163f2d4/volumes"
Oct 29 23:12:03.491741 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:03.491650    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"]
Oct 29 23:12:03.492246 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:03.491946    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz" podUID="95b92349-1ece-43d1-aa95-f4ca375813d0" containerName="manager" containerID="cri-o://0deeff0b1883775a1f5887b769cbde31582d1501679f3196216ea9fc7bcd1b5e" gracePeriod=10
Oct 29 23:12:03.749520 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:03.749452    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"
Oct 29 23:12:03.903986 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:03.903933    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b92349-1ece-43d1-aa95-f4ca375813d0-kube-api-access-jxrmw\" (UniqueName: \"kubernetes.io/projected/95b92349-1ece-43d1-aa95-f4ca375813d0-kube-api-access-jxrmw\") pod \"95b92349-1ece-43d1-aa95-f4ca375813d0\" (UID: \"95b92349-1ece-43d1-aa95-f4ca375813d0\") "
Oct 29 23:12:03.904240 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:03.904048    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b92349-1ece-43d1-aa95-f4ca375813d0-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95b92349-1ece-43d1-aa95-f4ca375813d0-extensions-socket-volume\") pod \"95b92349-1ece-43d1-aa95-f4ca375813d0\" (UID: \"95b92349-1ece-43d1-aa95-f4ca375813d0\") "
Oct 29 23:12:03.905084 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:03.905056    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b92349-1ece-43d1-aa95-f4ca375813d0-extensions-socket-volume" pod "95b92349-1ece-43d1-aa95-f4ca375813d0" (UID: "95b92349-1ece-43d1-aa95-f4ca375813d0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:12:03.906773 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:03.906750    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b92349-1ece-43d1-aa95-f4ca375813d0-kube-api-access-jxrmw" pod "95b92349-1ece-43d1-aa95-f4ca375813d0" (UID: "95b92349-1ece-43d1-aa95-f4ca375813d0"). InnerVolumeSpecName "kube-api-access-jxrmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:12:04.006085 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:04.005997    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxrmw\" (UniqueName: \"kubernetes.io/projected/95b92349-1ece-43d1-aa95-f4ca375813d0-kube-api-access-jxrmw\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:12:04.006085 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:04.006027    4559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95b92349-1ece-43d1-aa95-f4ca375813d0-extensions-socket-volume\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:12:04.447201 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:04.447164    4559 generic.go:356] "Generic (PLEG): container finished" podID="95b92349-1ece-43d1-aa95-f4ca375813d0" containerID="0deeff0b1883775a1f5887b769cbde31582d1501679f3196216ea9fc7bcd1b5e" exitCode=0
Oct 29 23:12:04.447389 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:04.447214    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"
Oct 29 23:12:04.447389 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:04.447249    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz" event={"ID":"95b92349-1ece-43d1-aa95-f4ca375813d0","Type":"ContainerDied","Data":"0deeff0b1883775a1f5887b769cbde31582d1501679f3196216ea9fc7bcd1b5e"}
Oct 29 23:12:04.447389 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:04.447312    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz" event={"ID":"95b92349-1ece-43d1-aa95-f4ca375813d0","Type":"ContainerDied","Data":"3b8375058f9c96cab7381d1b302cb29afdd655f123711ba8ad328accae005d5c"}
Oct 29 23:12:04.447389 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:04.447338    4559 scope.go:122] "RemoveContainer" containerID="0deeff0b1883775a1f5887b769cbde31582d1501679f3196216ea9fc7bcd1b5e"
Oct 29 23:12:04.457772 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:04.457747    4559 scope.go:122] "RemoveContainer" containerID="0deeff0b1883775a1f5887b769cbde31582d1501679f3196216ea9fc7bcd1b5e"
Oct 29 23:12:04.458106 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:12:04.458086    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0deeff0b1883775a1f5887b769cbde31582d1501679f3196216ea9fc7bcd1b5e\": container with ID starting with 0deeff0b1883775a1f5887b769cbde31582d1501679f3196216ea9fc7bcd1b5e not found: ID does not exist" containerID="0deeff0b1883775a1f5887b769cbde31582d1501679f3196216ea9fc7bcd1b5e"
Oct 29 23:12:04.458169 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:04.458120    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0deeff0b1883775a1f5887b769cbde31582d1501679f3196216ea9fc7bcd1b5e"} err="failed to get container status \"0deeff0b1883775a1f5887b769cbde31582d1501679f3196216ea9fc7bcd1b5e\": rpc error: code = NotFound desc = could not find container \"0deeff0b1883775a1f5887b769cbde31582d1501679f3196216ea9fc7bcd1b5e\": container with ID starting with 0deeff0b1883775a1f5887b769cbde31582d1501679f3196216ea9fc7bcd1b5e not found: ID does not exist"
Oct 29 23:12:04.512516 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:04.512478    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"]
Oct 29 23:12:04.525018 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:04.524981    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-wdqgz"]
Oct 29 23:12:05.225534 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:05.225488    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b92349-1ece-43d1-aa95-f4ca375813d0" path="/var/lib/kubelet/pods/95b92349-1ece-43d1-aa95-f4ca375813d0/volumes"
Oct 29 23:12:13.026782 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:13.026745    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"]
Oct 29 23:12:13.039316 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:13.039281    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dp4d7"]
Oct 29 23:12:13.226739 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:13.226701    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a4d57b5-7fd3-470c-8716-dc3053877b4e" path="/var/lib/kubelet/pods/3a4d57b5-7fd3-470c-8716-dc3053877b4e/volumes"
Oct 29 23:12:13.692978 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:12:13.692944    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:12:24.028652 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:24.028614    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"]
Oct 29 23:12:24.033684 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:24.033653    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bsj4f"]
Oct 29 23:12:25.226161 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:25.226122    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf46e142-9889-4fd2-b26c-4c14b9819bc0" path="/var/lib/kubelet/pods/bf46e142-9889-4fd2-b26c-4c14b9819bc0/volumes"
Oct 29 23:12:26.108009 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:26.107965    4559 scope.go:122] "RemoveContainer" containerID="67a253defcd0a5c59307c410eb3382d88c386ff2be866449ca34465e5cc0886b"
Oct 29 23:12:43.693375 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:12:43.693337    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:12:51.028611 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:51.028574    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"]
Oct 29 23:12:51.040796 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:51.040763    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gh4zf"]
Oct 29 23:12:51.226525 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:12:51.226485    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e665a4-a634-4eef-9ea1-ef3666aa47be" path="/var/lib/kubelet/pods/a8e665a4-a634-4eef-9ea1-ef3666aa47be/volumes"
Oct 29 23:13:01.026942 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:01.026907    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"]
Oct 29 23:13:01.039499 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:01.039461    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z7222"]
Oct 29 23:13:01.227499 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:01.227459    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75539855-d762-4253-bf8c-f12a53e26cbc" path="/var/lib/kubelet/pods/75539855-d762-4253-bf8c-f12a53e26cbc/volumes"
Oct 29 23:13:06.615130 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:06.615095    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p4nbv"]
Oct 29 23:13:06.615606 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:06.615503    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ef7d2cb2-e200-4c5b-82be-e778409fa251" containerName="revoke-keys"
Oct 29 23:13:06.615606 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:06.615523    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7d2cb2-e200-4c5b-82be-e778409fa251" containerName="revoke-keys"
Oct 29 23:13:06.615606 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:06.615550    4559 cpu_manager.go:414] "containerMap: removing container" podUID="95b92349-1ece-43d1-aa95-f4ca375813d0" containerName="manager"
Oct 29 23:13:06.615606 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:06.615557    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b92349-1ece-43d1-aa95-f4ca375813d0" containerName="manager"
Oct 29 23:13:06.615774 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:06.615644    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b92349-1ece-43d1-aa95-f4ca375813d0" containerName="manager"
Oct 29 23:13:06.615774 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:06.615658    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7d2cb2-e200-4c5b-82be-e778409fa251" containerName="revoke-keys"
Oct 29 23:13:06.619024 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:06.619005    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p4nbv"
Oct 29 23:13:06.633939 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:06.633909    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p4nbv"]
Oct 29 23:13:06.644532 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:06.644506    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6hs2c\""
Oct 29 23:13:06.719640 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:06.719604    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cswj\" (UniqueName: \"kubernetes.io/projected/67df7b6e-f634-4469-8c4f-eaf2ca607afe-kube-api-access-8cswj\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p4nbv\" (UID: \"67df7b6e-f634-4469-8c4f-eaf2ca607afe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p4nbv"
Oct 29 23:13:06.719844 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:06.719721    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67df7b6e-f634-4469-8c4f-eaf2ca607afe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p4nbv\" (UID: \"67df7b6e-f634-4469-8c4f-eaf2ca607afe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p4nbv"
Oct 29 23:13:06.820287 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:06.820250    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cswj\" (UniqueName: \"kubernetes.io/projected/67df7b6e-f634-4469-8c4f-eaf2ca607afe-kube-api-access-8cswj\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p4nbv\" (UID: \"67df7b6e-f634-4469-8c4f-eaf2ca607afe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p4nbv"
Oct 29 23:13:06.820505 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:06.820348    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67df7b6e-f634-4469-8c4f-eaf2ca607afe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p4nbv\" (UID: \"67df7b6e-f634-4469-8c4f-eaf2ca607afe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p4nbv"
Oct 29 23:13:06.820740 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:06.820719    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67df7b6e-f634-4469-8c4f-eaf2ca607afe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p4nbv\" (UID: \"67df7b6e-f634-4469-8c4f-eaf2ca607afe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p4nbv"
Oct 29 23:13:06.841086 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:06.841050    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cswj\" (UniqueName: \"kubernetes.io/projected/67df7b6e-f634-4469-8c4f-eaf2ca607afe-kube-api-access-8cswj\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p4nbv\" (UID: \"67df7b6e-f634-4469-8c4f-eaf2ca607afe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p4nbv"
Oct 29 23:13:06.933371 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:06.933330    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p4nbv"
Oct 29 23:13:07.272289 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:07.272263    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p4nbv"]
Oct 29 23:13:07.274337 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:13:07.274307    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67df7b6e_f634_4469_8c4f_eaf2ca607afe.slice/crio-b0c55b34e76252ac171e997fdfd9ad9142f5dfb93a9c9e8b86f9316fe73325c4 WatchSource:0}: Error finding container b0c55b34e76252ac171e997fdfd9ad9142f5dfb93a9c9e8b86f9316fe73325c4: Status 404 returned error can't find the container with id b0c55b34e76252ac171e997fdfd9ad9142f5dfb93a9c9e8b86f9316fe73325c4
Oct 29 23:13:07.765790 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:07.765743    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p4nbv" event={"ID":"67df7b6e-f634-4469-8c4f-eaf2ca607afe","Type":"ContainerStarted","Data":"49926dfe0e71af7f8fec5efb976a5721a5fa4c154003e26b44ad8e73e2856883"}
Oct 29 23:13:07.766330 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:07.765874    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p4nbv" event={"ID":"67df7b6e-f634-4469-8c4f-eaf2ca607afe","Type":"ContainerStarted","Data":"b0c55b34e76252ac171e997fdfd9ad9142f5dfb93a9c9e8b86f9316fe73325c4"}
Oct 29 23:13:07.766330 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:07.766126    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p4nbv"
Oct 29 23:13:07.816235 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:07.816176    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p4nbv" podStartSLOduration=1.816159994 podStartE2EDuration="1.816159994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:07.81361938 +0000 UTC m=+3526.725062933" watchObservedRunningTime="2026-10-29 23:13:07.816159994 +0000 UTC m=+3526.727603547"
Oct 29 23:13:13.692942 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:13:13.692904    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:13:18.773759 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:18.773728    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p4nbv"
Oct 29 23:13:24.983691 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:24.983644    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:13:26.144406 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:26.144364    4559 scope.go:122] "RemoveContainer" containerID="749ceee12ba636af3c09ddd774303593312ed5e8f915aa8bbb410c248fdcf151"
Oct 29 23:13:26.157293 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:26.157256    4559 scope.go:122] "RemoveContainer" containerID="403cb63076b6fe1fb65d7e61042c58cffb76fbf04de00e95acdf0f7d1925cf9f"
Oct 29 23:13:26.169467 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:26.169424    4559 scope.go:122] "RemoveContainer" containerID="a550812727bef3aa2180340ded4dde508c4c26fe22dabb007eeb07abb3c06c0c"
Oct 29 23:13:40.114614 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:40.114516    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:13:43.709263 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:13:43.709217    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:13:45.083836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:45.077115    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:13:45.083836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:13:45.077189    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg" podUID="5716b06a-0198-4647-884c-3251a6a4b63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:01.053202 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.053159    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"]
Oct 29 23:14:01.057852 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.057798    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:01.074497 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.074467    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-063169-serving-cert\""
Oct 29 23:14:01.087199 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.087163    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:14:01.087408 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.087287    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:14:01.092765 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.092732    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"]
Oct 29 23:14:01.140277 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.140237    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dzwhk\""
Oct 29 23:14:01.140563 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.140286    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 23:14:01.151110 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.151081    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz\" (UID: \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:01.151269 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.151210    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz\" (UID: \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:01.151269 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.151233    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz\" (UID: \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:01.151350 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.151295    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrzqr\" (UniqueName: \"kubernetes.io/projected/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-kube-api-access-hrzqr\") pod \"maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz\" (UID: \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:01.252303 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.252258    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrzqr\" (UniqueName: \"kubernetes.io/projected/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-kube-api-access-hrzqr\") pod \"maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz\" (UID: \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:01.252632 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.252611    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz\" (UID: \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:01.253628 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.253597    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz\" (UID: \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:01.254037 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.254009    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz\" (UID: \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:01.255713 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.255680    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz\" (UID: \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:01.263199 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.263162    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz\" (UID: \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:01.265259 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.265230    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz\" (UID: \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:01.290114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.290073    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrzqr\" (UniqueName: \"kubernetes.io/projected/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-kube-api-access-hrzqr\") pod \"maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz\" (UID: \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:01.372928 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.372820    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:01.673184 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:01.673157    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"]
Oct 29 23:14:01.676006 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:14:01.675982    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f49e88_6b39_4c79_9de6_3a385b0a88e5.slice/crio-49051780826c8f22718c1797a09193dbeddada3fe4e65815d50755dd51c45ed7 WatchSource:0}: Error finding container 49051780826c8f22718c1797a09193dbeddada3fe4e65815d50755dd51c45ed7: Status 404 returned error can't find the container with id 49051780826c8f22718c1797a09193dbeddada3fe4e65815d50755dd51c45ed7
Oct 29 23:14:02.303360 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:02.303315    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz" event={"ID":"c9f49e88-6b39-4c79-9de6-3a385b0a88e5","Type":"ContainerStarted","Data":"c41425081c7b38803df5d4a4f565d3e77bffddee79f32d834453506b8d7fddb4"}
Oct 29 23:14:02.304066 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:02.304037    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz" event={"ID":"c9f49e88-6b39-4c79-9de6-3a385b0a88e5","Type":"ContainerStarted","Data":"49051780826c8f22718c1797a09193dbeddada3fe4e65815d50755dd51c45ed7"}
Oct 29 23:14:02.305276 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:02.305239    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:02.347499 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:02.347376    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz" podStartSLOduration=0.943961296 podStartE2EDuration="1.347359595s" totalImagesPullingTime="403.398299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:02.346534091 +0000 UTC m=+3581.257977648" watchObservedRunningTime="2026-10-29 23:14:02.347359595 +0000 UTC m=+3581.258803149"
Oct 29 23:14:04.723954 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:04.723901    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:14:09.320037 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:09.320003    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:13.694094 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:14:13.694056    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:14:20.084380 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:20.084308    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:14:21.981406 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:21.981365    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 23:14:21.982998 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:21.982975    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 23:14:26.271463 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:26.271429    4559 scope.go:122] "RemoveContainer" containerID="42996a27f545acab001a3a0529201e5560c807a7b0510ef1766596c472ba583a"
Oct 29 23:14:26.280176 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:26.280154    4559 scope.go:122] "RemoveContainer" containerID="43dc83bf387ede5ffa52381fe7737322bf731fa825cdb6d152db2291639566b4"
Oct 29 23:14:34.006972 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:34.006926    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:14:43.693341 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:14:43.693300    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:14:45.291271 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:45.291232    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:14:57.867225 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:57.867184    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"]
Oct 29 23:14:57.868128 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:57.867448    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz" podUID="c9f49e88-6b39-4c79-9de6-3a385b0a88e5" containerName="maas-api" containerID="cri-o://c41425081c7b38803df5d4a4f565d3e77bffddee79f32d834453506b8d7fddb4" gracePeriod=30
Oct 29 23:14:58.370513 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.370486    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:58.480127 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.480024    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-maas-api-tls\") pod \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\" (UID: \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\") "
Oct 29 23:14:58.480127 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.480088    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-metrics-certs\") pod \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\" (UID: \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\") "
Oct 29 23:14:58.480342 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.480240    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-openshift-service-ca\") pod \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\" (UID: \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\") "
Oct 29 23:14:58.480342 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.480292    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-kube-api-access-hrzqr\" (UniqueName: \"kubernetes.io/projected/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-kube-api-access-hrzqr\") pod \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\" (UID: \"c9f49e88-6b39-4c79-9de6-3a385b0a88e5\") "
Oct 29 23:14:58.480701 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.480658    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-openshift-service-ca" pod "c9f49e88-6b39-4c79-9de6-3a385b0a88e5" (UID: "c9f49e88-6b39-4c79-9de6-3a385b0a88e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:14:58.482461 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.482433    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-maas-api-tls" pod "c9f49e88-6b39-4c79-9de6-3a385b0a88e5" (UID: "c9f49e88-6b39-4c79-9de6-3a385b0a88e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:58.482571 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.482497    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-metrics-certs" pod "c9f49e88-6b39-4c79-9de6-3a385b0a88e5" (UID: "c9f49e88-6b39-4c79-9de6-3a385b0a88e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:58.482571 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.482553    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-kube-api-access-hrzqr" pod "c9f49e88-6b39-4c79-9de6-3a385b0a88e5" (UID: "c9f49e88-6b39-4c79-9de6-3a385b0a88e5"). InnerVolumeSpecName "kube-api-access-hrzqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:14:58.582066 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.582019    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrzqr\" (UniqueName: \"kubernetes.io/projected/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-kube-api-access-hrzqr\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:14:58.582066 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.582066    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-maas-api-tls\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:14:58.582310 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.582079    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-metrics-certs\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:14:58.582310 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.582090    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f49e88-6b39-4c79-9de6-3a385b0a88e5-openshift-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:14:58.592590 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.592547    4559 generic.go:356] "Generic (PLEG): container finished" podID="c9f49e88-6b39-4c79-9de6-3a385b0a88e5" containerID="c41425081c7b38803df5d4a4f565d3e77bffddee79f32d834453506b8d7fddb4" exitCode=0
Oct 29 23:14:58.592805 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.592595    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"
Oct 29 23:14:58.592805 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.592637    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz" event={"ID":"c9f49e88-6b39-4c79-9de6-3a385b0a88e5","Type":"ContainerDied","Data":"c41425081c7b38803df5d4a4f565d3e77bffddee79f32d834453506b8d7fddb4"}
Oct 29 23:14:58.592805 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.592691    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz" event={"ID":"c9f49e88-6b39-4c79-9de6-3a385b0a88e5","Type":"ContainerDied","Data":"49051780826c8f22718c1797a09193dbeddada3fe4e65815d50755dd51c45ed7"}
Oct 29 23:14:58.592805 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.592714    4559 scope.go:122] "RemoveContainer" containerID="c41425081c7b38803df5d4a4f565d3e77bffddee79f32d834453506b8d7fddb4"
Oct 29 23:14:58.604013 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.603987    4559 scope.go:122] "RemoveContainer" containerID="c41425081c7b38803df5d4a4f565d3e77bffddee79f32d834453506b8d7fddb4"
Oct 29 23:14:58.604328 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:14:58.604308    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c41425081c7b38803df5d4a4f565d3e77bffddee79f32d834453506b8d7fddb4\": container with ID starting with c41425081c7b38803df5d4a4f565d3e77bffddee79f32d834453506b8d7fddb4 not found: ID does not exist" containerID="c41425081c7b38803df5d4a4f565d3e77bffddee79f32d834453506b8d7fddb4"
Oct 29 23:14:58.604393 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.604337    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41425081c7b38803df5d4a4f565d3e77bffddee79f32d834453506b8d7fddb4"} err="failed to get container status \"c41425081c7b38803df5d4a4f565d3e77bffddee79f32d834453506b8d7fddb4\": rpc error: code = NotFound desc = could not find container \"c41425081c7b38803df5d4a4f565d3e77bffddee79f32d834453506b8d7fddb4\": container with ID starting with c41425081c7b38803df5d4a4f565d3e77bffddee79f32d834453506b8d7fddb4 not found: ID does not exist"
Oct 29 23:14:58.648651 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.648620    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"]
Oct 29 23:14:58.651767 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:58.651735    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-063169-cf6bf8694-f8vgz"]
Oct 29 23:14:59.226410 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:59.226376    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f49e88-6b39-4c79-9de6-3a385b0a88e5" path="/var/lib/kubelet/pods/c9f49e88-6b39-4c79-9de6-3a385b0a88e5/volumes"
Oct 29 23:14:59.319941 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:14:59.319896    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:15:00.142983 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:00.142942    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8"]
Oct 29 23:15:00.143601 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:00.143580    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c9f49e88-6b39-4c79-9de6-3a385b0a88e5" containerName="maas-api"
Oct 29 23:15:00.143652 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:00.143604    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f49e88-6b39-4c79-9de6-3a385b0a88e5" containerName="maas-api"
Oct 29 23:15:00.143718 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:00.143706    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f49e88-6b39-4c79-9de6-3a385b0a88e5" containerName="maas-api"
Oct 29 23:15:00.148633 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:00.148607    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8"
Oct 29 23:15:00.166420 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:00.166387    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8"]
Oct 29 23:15:00.176246 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:00.176217    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:15:00.205606 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:00.205563    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm6wc\" (UniqueName: \"kubernetes.io/projected/84f0ecda-26a7-4d41-b4aa-ddd64da45bd9-kube-api-access-bm6wc\") pod \"maas-api-key-cleanup-29888595-jrpk8\" (UID: \"84f0ecda-26a7-4d41-b4aa-ddd64da45bd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8"
Oct 29 23:15:00.205778 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:00.205725    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f0ecda-26a7-4d41-b4aa-ddd64da45bd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888595-jrpk8\" (UID: \"84f0ecda-26a7-4d41-b4aa-ddd64da45bd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8"
Oct 29 23:15:00.214440 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:00.214414    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:15:00.214545 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:00.214421    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:15:00.307070 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:00.307027    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f0ecda-26a7-4d41-b4aa-ddd64da45bd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888595-jrpk8\" (UID: \"84f0ecda-26a7-4d41-b4aa-ddd64da45bd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8"
Oct 29 23:15:00.307663 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:00.307238    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm6wc\" (UniqueName: \"kubernetes.io/projected/84f0ecda-26a7-4d41-b4aa-ddd64da45bd9-kube-api-access-bm6wc\") pod \"maas-api-key-cleanup-29888595-jrpk8\" (UID: \"84f0ecda-26a7-4d41-b4aa-ddd64da45bd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8"
Oct 29 23:15:00.307893 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:00.307873    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f0ecda-26a7-4d41-b4aa-ddd64da45bd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888595-jrpk8\" (UID: \"84f0ecda-26a7-4d41-b4aa-ddd64da45bd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8"
Oct 29 23:15:00.327450 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:00.327423    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm6wc\" (UniqueName: \"kubernetes.io/projected/84f0ecda-26a7-4d41-b4aa-ddd64da45bd9-kube-api-access-bm6wc\") pod \"maas-api-key-cleanup-29888595-jrpk8\" (UID: \"84f0ecda-26a7-4d41-b4aa-ddd64da45bd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8"
Oct 29 23:15:00.465323 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:00.465235    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8"
Oct 29 23:15:00.770953 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:00.770921    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8"]
Oct 29 23:15:00.773102 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:15:00.773070    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f0ecda_26a7_4d41_b4aa_ddd64da45bd9.slice/crio-1c1fa1f5d96530be75a2771a094f471ca6c2c1f03d4d26a13e7210d8cab98dea WatchSource:0}: Error finding container 1c1fa1f5d96530be75a2771a094f471ca6c2c1f03d4d26a13e7210d8cab98dea: Status 404 returned error can't find the container with id 1c1fa1f5d96530be75a2771a094f471ca6c2c1f03d4d26a13e7210d8cab98dea
Oct 29 23:15:01.610101 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:01.610065    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8" event={"ID":"84f0ecda-26a7-4d41-b4aa-ddd64da45bd9","Type":"ContainerStarted","Data":"fc39e195d036f8089e0b2e22b567f163ae2e38ab83a5d551cfeeffaccf1ca7aa"}
Oct 29 23:15:01.610531 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:01.610117    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8" event={"ID":"84f0ecda-26a7-4d41-b4aa-ddd64da45bd9","Type":"ContainerStarted","Data":"1c1fa1f5d96530be75a2771a094f471ca6c2c1f03d4d26a13e7210d8cab98dea"}
Oct 29 23:15:01.649461 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:01.649404    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8" podStartSLOduration=1.6493869 podStartE2EDuration="1.6493869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:01.648318801 +0000 UTC m=+3640.559762356" watchObservedRunningTime="2026-10-29 23:15:01.6493869 +0000 UTC m=+3640.560830455"
Oct 29 23:15:02.617221 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:02.617135    4559 generic.go:356] "Generic (PLEG): container finished" podID="84f0ecda-26a7-4d41-b4aa-ddd64da45bd9" containerID="fc39e195d036f8089e0b2e22b567f163ae2e38ab83a5d551cfeeffaccf1ca7aa" exitCode=60
Oct 29 23:15:02.617628 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:02.617220    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8" event={"ID":"84f0ecda-26a7-4d41-b4aa-ddd64da45bd9","Type":"ContainerDied","Data":"fc39e195d036f8089e0b2e22b567f163ae2e38ab83a5d551cfeeffaccf1ca7aa"}
Oct 29 23:15:02.617628 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:02.617454    4559 scope.go:122] "RemoveContainer" containerID="fc39e195d036f8089e0b2e22b567f163ae2e38ab83a5d551cfeeffaccf1ca7aa"
Oct 29 23:15:03.623752 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:03.623703    4559 generic.go:356] "Generic (PLEG): container finished" podID="84f0ecda-26a7-4d41-b4aa-ddd64da45bd9" containerID="8841b638b836323d308c39bbd0659b6f3d1c53aed7cf16bf2286475e7a34571b" exitCode=60
Oct 29 23:15:03.624263 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:03.623794    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8" event={"ID":"84f0ecda-26a7-4d41-b4aa-ddd64da45bd9","Type":"ContainerDied","Data":"8841b638b836323d308c39bbd0659b6f3d1c53aed7cf16bf2286475e7a34571b"}
Oct 29 23:15:03.624263 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:03.623877    4559 scope.go:122] "RemoveContainer" containerID="fc39e195d036f8089e0b2e22b567f163ae2e38ab83a5d551cfeeffaccf1ca7aa"
Oct 29 23:15:03.624263 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:03.624154    4559 scope.go:122] "RemoveContainer" containerID="8841b638b836323d308c39bbd0659b6f3d1c53aed7cf16bf2286475e7a34571b"
Oct 29 23:15:03.624459 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:15:03.624439    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888595-jrpk8_odh-ai-gateway-infra(84f0ecda-26a7-4d41-b4aa-ddd64da45bd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8" podUID="84f0ecda-26a7-4d41-b4aa-ddd64da45bd9"
Oct 29 23:15:04.630496 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:04.630464    4559 scope.go:122] "RemoveContainer" containerID="8841b638b836323d308c39bbd0659b6f3d1c53aed7cf16bf2286475e7a34571b"
Oct 29 23:15:04.631033 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:15:04.630780    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888595-jrpk8_odh-ai-gateway-infra(84f0ecda-26a7-4d41-b4aa-ddd64da45bd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8" podUID="84f0ecda-26a7-4d41-b4aa-ddd64da45bd9"
Oct 29 23:15:13.019343 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:13.019305    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:15:13.707381 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:15:13.707317    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:15:14.220556 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:14.220513    4559 scope.go:122] "RemoveContainer" containerID="8841b638b836323d308c39bbd0659b6f3d1c53aed7cf16bf2286475e7a34571b"
Oct 29 23:15:14.735846 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:14.732944    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8" event={"ID":"84f0ecda-26a7-4d41-b4aa-ddd64da45bd9","Type":"ContainerStarted","Data":"f4ced8a5d3bbd6887976997e8d2270d3f6b43f57dda99cd53ff7bbb08de6b282"}
Oct 29 23:15:15.766907 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:15.765152    4559 generic.go:356] "Generic (PLEG): container finished" podID="84f0ecda-26a7-4d41-b4aa-ddd64da45bd9" containerID="f4ced8a5d3bbd6887976997e8d2270d3f6b43f57dda99cd53ff7bbb08de6b282" exitCode=60
Oct 29 23:15:15.766907 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:15.765237    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8" event={"ID":"84f0ecda-26a7-4d41-b4aa-ddd64da45bd9","Type":"ContainerDied","Data":"f4ced8a5d3bbd6887976997e8d2270d3f6b43f57dda99cd53ff7bbb08de6b282"}
Oct 29 23:15:15.766907 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:15.765302    4559 scope.go:122] "RemoveContainer" containerID="8841b638b836323d308c39bbd0659b6f3d1c53aed7cf16bf2286475e7a34571b"
Oct 29 23:15:15.766907 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:15.765800    4559 scope.go:122] "RemoveContainer" containerID="f4ced8a5d3bbd6887976997e8d2270d3f6b43f57dda99cd53ff7bbb08de6b282"
Oct 29 23:15:15.766907 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:15:15.766146    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888595-jrpk8_odh-ai-gateway-infra(84f0ecda-26a7-4d41-b4aa-ddd64da45bd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8" podUID="84f0ecda-26a7-4d41-b4aa-ddd64da45bd9"
Oct 29 23:15:15.806009 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:15.805967    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8"]
Oct 29 23:15:17.234905 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:17.234872    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8"
Oct 29 23:15:17.320353 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:17.320313    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f0ecda-26a7-4d41-b4aa-ddd64da45bd9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f0ecda-26a7-4d41-b4aa-ddd64da45bd9-maas-api-service-ca\") pod \"84f0ecda-26a7-4d41-b4aa-ddd64da45bd9\" (UID: \"84f0ecda-26a7-4d41-b4aa-ddd64da45bd9\") "
Oct 29 23:15:17.320543 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:17.320381    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f0ecda-26a7-4d41-b4aa-ddd64da45bd9-kube-api-access-bm6wc\" (UniqueName: \"kubernetes.io/projected/84f0ecda-26a7-4d41-b4aa-ddd64da45bd9-kube-api-access-bm6wc\") pod \"84f0ecda-26a7-4d41-b4aa-ddd64da45bd9\" (UID: \"84f0ecda-26a7-4d41-b4aa-ddd64da45bd9\") "
Oct 29 23:15:17.320738 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:17.320710    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f0ecda-26a7-4d41-b4aa-ddd64da45bd9-maas-api-service-ca" pod "84f0ecda-26a7-4d41-b4aa-ddd64da45bd9" (UID: "84f0ecda-26a7-4d41-b4aa-ddd64da45bd9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:17.322654 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:17.322631    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f0ecda-26a7-4d41-b4aa-ddd64da45bd9-kube-api-access-bm6wc" pod "84f0ecda-26a7-4d41-b4aa-ddd64da45bd9" (UID: "84f0ecda-26a7-4d41-b4aa-ddd64da45bd9"). InnerVolumeSpecName "kube-api-access-bm6wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:17.421766 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:17.421731    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f0ecda-26a7-4d41-b4aa-ddd64da45bd9-maas-api-service-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:15:17.421766 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:17.421760    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm6wc\" (UniqueName: \"kubernetes.io/projected/84f0ecda-26a7-4d41-b4aa-ddd64da45bd9-kube-api-access-bm6wc\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:15:17.797983 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:17.797878    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8"
Oct 29 23:15:17.797983 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:17.797919    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8" event={"ID":"84f0ecda-26a7-4d41-b4aa-ddd64da45bd9","Type":"ContainerDied","Data":"1c1fa1f5d96530be75a2771a094f471ca6c2c1f03d4d26a13e7210d8cab98dea"}
Oct 29 23:15:17.798232 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:17.797995    4559 scope.go:122] "RemoveContainer" containerID="f4ced8a5d3bbd6887976997e8d2270d3f6b43f57dda99cd53ff7bbb08de6b282"
Oct 29 23:15:17.855748 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:17.855703    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8"]
Oct 29 23:15:17.869354 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:17.869319    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-jrpk8"]
Oct 29 23:15:19.226084 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:19.226048    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84f0ecda-26a7-4d41-b4aa-ddd64da45bd9" path="/var/lib/kubelet/pods/84f0ecda-26a7-4d41-b4aa-ddd64da45bd9/volumes"
Oct 29 23:15:28.505445 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:28.505404    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:15:43.693585 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:15:43.693543    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:15:44.413932 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:44.413895    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:15:45.581919 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.581861    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc"]
Oct 29 23:15:45.582501 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.582486    4559 cpu_manager.go:414] "containerMap: removing container" podUID="84f0ecda-26a7-4d41-b4aa-ddd64da45bd9" containerName="cleanup"
Oct 29 23:15:45.582582 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.582505    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f0ecda-26a7-4d41-b4aa-ddd64da45bd9" containerName="cleanup"
Oct 29 23:15:45.582582 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.582523    4559 cpu_manager.go:414] "containerMap: removing container" podUID="84f0ecda-26a7-4d41-b4aa-ddd64da45bd9" containerName="cleanup"
Oct 29 23:15:45.582582 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.582532    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f0ecda-26a7-4d41-b4aa-ddd64da45bd9" containerName="cleanup"
Oct 29 23:15:45.582760 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.582694    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f0ecda-26a7-4d41-b4aa-ddd64da45bd9" containerName="cleanup"
Oct 29 23:15:45.582760 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.582709    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f0ecda-26a7-4d41-b4aa-ddd64da45bd9" containerName="cleanup"
Oct 29 23:15:45.586634 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.586607    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc"
Oct 29 23:15:45.606557 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.606525    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc"]
Oct 29 23:15:45.618476 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.618433    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:15:45.619035 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.618856    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:15:45.619216 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.618946    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:15:45.619363 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.618981    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:15:45.693088 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.693034    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5zbt\" (UniqueName: \"kubernetes.io/projected/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-kube-api-access-q5zbt\") pod \"maas-api-revoke-sub-452b6d7b-4cmqc\" (UID: \"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc"
Oct 29 23:15:45.693318 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.693151    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4cmqc\" (UID: \"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc"
Oct 29 23:15:45.693318 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.693221    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4cmqc\" (UID: \"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc"
Oct 29 23:15:45.794095 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.794050    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4cmqc\" (UID: \"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc"
Oct 29 23:15:45.794342 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.794154    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4cmqc\" (UID: \"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc"
Oct 29 23:15:45.794342 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.794244    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5zbt\" (UniqueName: \"kubernetes.io/projected/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-kube-api-access-q5zbt\") pod \"maas-api-revoke-sub-452b6d7b-4cmqc\" (UID: \"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc"
Oct 29 23:15:45.795484 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.795452    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4cmqc\" (UID: \"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc"
Oct 29 23:15:45.810940 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.810899    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4cmqc\" (UID: \"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc"
Oct 29 23:15:45.819630 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.819591    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5zbt\" (UniqueName: \"kubernetes.io/projected/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-kube-api-access-q5zbt\") pod \"maas-api-revoke-sub-452b6d7b-4cmqc\" (UID: \"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc"
Oct 29 23:15:45.911964 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:45.911905    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc"
Oct 29 23:15:46.511837 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:46.511718    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc"]
Oct 29 23:15:46.583585 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:15:46.583549    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf4495d_9d0f_4bef_a1a6_d9823cd4def5.slice/crio-1c2a48fb23423dd0cdbb902459d1398e95570c2b19ce3a4272b7f1db0770ce3f WatchSource:0}: Error finding container 1c2a48fb23423dd0cdbb902459d1398e95570c2b19ce3a4272b7f1db0770ce3f: Status 404 returned error can't find the container with id 1c2a48fb23423dd0cdbb902459d1398e95570c2b19ce3a4272b7f1db0770ce3f
Oct 29 23:15:46.992865 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:46.991275    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc" event={"ID":"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5","Type":"ContainerStarted","Data":"1c2a48fb23423dd0cdbb902459d1398e95570c2b19ce3a4272b7f1db0770ce3f"}
Oct 29 23:15:48.159836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:48.159696    4559 generic.go:356] "Generic (PLEG): container finished" podID="1bf4495d-9d0f-4bef-a1a6-d9823cd4def5" containerID="733aea41d7da6f0187c429ffef2ad236b8d8b4495f090bd0aba6db764b428e16" exitCode=0
Oct 29 23:15:48.160414 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:48.159965    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc" event={"ID":"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5","Type":"ContainerDied","Data":"733aea41d7da6f0187c429ffef2ad236b8d8b4495f090bd0aba6db764b428e16"}
Oct 29 23:15:49.306999 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:49.306971    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc"
Oct 29 23:15:49.385571 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:49.385532    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-kube-api-access-q5zbt\" (UniqueName: \"kubernetes.io/projected/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-kube-api-access-q5zbt\") pod \"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5\" (UID: \"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5\") "
Oct 29 23:15:49.385751 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:49.385612    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-maas-api-openshift-ca\") pod \"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5\" (UID: \"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5\") "
Oct 29 23:15:49.385751 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:49.385672    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-maas-api-xks-ca\") pod \"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5\" (UID: \"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5\") "
Oct 29 23:15:49.386150 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:49.386125    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-maas-api-openshift-ca" pod "1bf4495d-9d0f-4bef-a1a6-d9823cd4def5" (UID: "1bf4495d-9d0f-4bef-a1a6-d9823cd4def5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:49.387928 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:49.387895    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-kube-api-access-q5zbt" pod "1bf4495d-9d0f-4bef-a1a6-d9823cd4def5" (UID: "1bf4495d-9d0f-4bef-a1a6-d9823cd4def5"). InnerVolumeSpecName "kube-api-access-q5zbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:49.388062 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:49.388027    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-maas-api-xks-ca" pod "1bf4495d-9d0f-4bef-a1a6-d9823cd4def5" (UID: "1bf4495d-9d0f-4bef-a1a6-d9823cd4def5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:15:49.486875 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:49.486767    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:15:49.486875 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:49.486805    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5zbt\" (UniqueName: \"kubernetes.io/projected/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-kube-api-access-q5zbt\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:15:49.486875 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:49.486845    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf4495d-9d0f-4bef-a1a6-d9823cd4def5-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:15:50.170829 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:50.170786    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc"
Oct 29 23:15:50.171036 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:50.170845    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cmqc" event={"ID":"1bf4495d-9d0f-4bef-a1a6-d9823cd4def5","Type":"ContainerDied","Data":"1c2a48fb23423dd0cdbb902459d1398e95570c2b19ce3a4272b7f1db0770ce3f"}
Oct 29 23:15:50.171036 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:50.170906    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c2a48fb23423dd0cdbb902459d1398e95570c2b19ce3a4272b7f1db0770ce3f"
Oct 29 23:15:58.713154 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:15:58.713112    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:16:04.388839 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.388770    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:16:04.669036 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.669002    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24"]
Oct 29 23:16:04.669429 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.669414    4559 cpu_manager.go:414] "containerMap: removing container" podUID="1bf4495d-9d0f-4bef-a1a6-d9823cd4def5" containerName="revoke-keys"
Oct 29 23:16:04.669491 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.669431    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf4495d-9d0f-4bef-a1a6-d9823cd4def5" containerName="revoke-keys"
Oct 29 23:16:04.669491 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.669470    4559 cpu_manager.go:414] "containerMap: removing container" podUID="84f0ecda-26a7-4d41-b4aa-ddd64da45bd9" containerName="cleanup"
Oct 29 23:16:04.669491 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.669475    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f0ecda-26a7-4d41-b4aa-ddd64da45bd9" containerName="cleanup"
Oct 29 23:16:04.669590 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.669540    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f0ecda-26a7-4d41-b4aa-ddd64da45bd9" containerName="cleanup"
Oct 29 23:16:04.669590 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.669551    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf4495d-9d0f-4bef-a1a6-d9823cd4def5" containerName="revoke-keys"
Oct 29 23:16:04.672754 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.672734    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24"
Oct 29 23:16:04.693133 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.693094    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24"]
Oct 29 23:16:04.703058 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.703025    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:16:04.703325 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.703296    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:16:04.703435 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.703379    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:16:04.703497 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.703441    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:16:04.709269 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.709240    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0537e8ab-6bd2-43df-b07f-e0690bf60cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-l4m24\" (UID: \"0537e8ab-6bd2-43df-b07f-e0690bf60cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24"
Oct 29 23:16:04.709425 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.709297    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prldg\" (UniqueName: \"kubernetes.io/projected/0537e8ab-6bd2-43df-b07f-e0690bf60cca-kube-api-access-prldg\") pod \"maas-api-revoke-sub-3e31a506-l4m24\" (UID: \"0537e8ab-6bd2-43df-b07f-e0690bf60cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24"
Oct 29 23:16:04.709535 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.709509    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0537e8ab-6bd2-43df-b07f-e0690bf60cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-l4m24\" (UID: \"0537e8ab-6bd2-43df-b07f-e0690bf60cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24"
Oct 29 23:16:04.811597 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.811547    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0537e8ab-6bd2-43df-b07f-e0690bf60cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-l4m24\" (UID: \"0537e8ab-6bd2-43df-b07f-e0690bf60cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24"
Oct 29 23:16:04.812655 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.812621    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0537e8ab-6bd2-43df-b07f-e0690bf60cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-l4m24\" (UID: \"0537e8ab-6bd2-43df-b07f-e0690bf60cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24"
Oct 29 23:16:04.812981 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.812958    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0537e8ab-6bd2-43df-b07f-e0690bf60cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-l4m24\" (UID: \"0537e8ab-6bd2-43df-b07f-e0690bf60cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24"
Oct 29 23:16:04.813157 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.813140    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prldg\" (UniqueName: \"kubernetes.io/projected/0537e8ab-6bd2-43df-b07f-e0690bf60cca-kube-api-access-prldg\") pod \"maas-api-revoke-sub-3e31a506-l4m24\" (UID: \"0537e8ab-6bd2-43df-b07f-e0690bf60cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24"
Oct 29 23:16:04.817116 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.817062    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0537e8ab-6bd2-43df-b07f-e0690bf60cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-l4m24\" (UID: \"0537e8ab-6bd2-43df-b07f-e0690bf60cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24"
Oct 29 23:16:04.858862 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.851681    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prldg\" (UniqueName: \"kubernetes.io/projected/0537e8ab-6bd2-43df-b07f-e0690bf60cca-kube-api-access-prldg\") pod \"maas-api-revoke-sub-3e31a506-l4m24\" (UID: \"0537e8ab-6bd2-43df-b07f-e0690bf60cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24"
Oct 29 23:16:04.992250 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:04.992132    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24"
Oct 29 23:16:05.348246 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:05.347053    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24"]
Oct 29 23:16:05.363566 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:16:05.352635    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0537e8ab_6bd2_43df_b07f_e0690bf60cca.slice/crio-28d47b9b6ffc64ad4ec73f4da0ae81c22ad4254011fa79576c389ad8a3255955 WatchSource:0}: Error finding container 28d47b9b6ffc64ad4ec73f4da0ae81c22ad4254011fa79576c389ad8a3255955: Status 404 returned error can't find the container with id 28d47b9b6ffc64ad4ec73f4da0ae81c22ad4254011fa79576c389ad8a3255955
Oct 29 23:16:05.406461 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:05.406423    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24" event={"ID":"0537e8ab-6bd2-43df-b07f-e0690bf60cca","Type":"ContainerStarted","Data":"28d47b9b6ffc64ad4ec73f4da0ae81c22ad4254011fa79576c389ad8a3255955"}
Oct 29 23:16:06.431836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:06.429394    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24" event={"ID":"0537e8ab-6bd2-43df-b07f-e0690bf60cca","Type":"ContainerStarted","Data":"e2404b5171b1a83aa74bd23708ab5e82e80acbf2de6e95c43a225f9c21137f46"}
Oct 29 23:16:07.481841 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:07.480839    4559 generic.go:356] "Generic (PLEG): container finished" podID="0537e8ab-6bd2-43df-b07f-e0690bf60cca" containerID="e2404b5171b1a83aa74bd23708ab5e82e80acbf2de6e95c43a225f9c21137f46" exitCode=0
Oct 29 23:16:07.481841 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:07.481261    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24" event={"ID":"0537e8ab-6bd2-43df-b07f-e0690bf60cca","Type":"ContainerDied","Data":"e2404b5171b1a83aa74bd23708ab5e82e80acbf2de6e95c43a225f9c21137f46"}
Oct 29 23:16:08.643937 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:08.643906    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24"
Oct 29 23:16:08.694453 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:08.694414    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0537e8ab-6bd2-43df-b07f-e0690bf60cca-kube-api-access-prldg\" (UniqueName: \"kubernetes.io/projected/0537e8ab-6bd2-43df-b07f-e0690bf60cca-kube-api-access-prldg\") pod \"0537e8ab-6bd2-43df-b07f-e0690bf60cca\" (UID: \"0537e8ab-6bd2-43df-b07f-e0690bf60cca\") "
Oct 29 23:16:08.694650 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:08.694473    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0537e8ab-6bd2-43df-b07f-e0690bf60cca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0537e8ab-6bd2-43df-b07f-e0690bf60cca-maas-api-xks-ca\") pod \"0537e8ab-6bd2-43df-b07f-e0690bf60cca\" (UID: \"0537e8ab-6bd2-43df-b07f-e0690bf60cca\") "
Oct 29 23:16:08.694650 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:08.694564    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0537e8ab-6bd2-43df-b07f-e0690bf60cca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0537e8ab-6bd2-43df-b07f-e0690bf60cca-maas-api-openshift-ca\") pod \"0537e8ab-6bd2-43df-b07f-e0690bf60cca\" (UID: \"0537e8ab-6bd2-43df-b07f-e0690bf60cca\") "
Oct 29 23:16:08.695021 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:08.694984    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0537e8ab-6bd2-43df-b07f-e0690bf60cca-maas-api-openshift-ca" pod "0537e8ab-6bd2-43df-b07f-e0690bf60cca" (UID: "0537e8ab-6bd2-43df-b07f-e0690bf60cca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:16:08.696803 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:08.696777    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0537e8ab-6bd2-43df-b07f-e0690bf60cca-kube-api-access-prldg" pod "0537e8ab-6bd2-43df-b07f-e0690bf60cca" (UID: "0537e8ab-6bd2-43df-b07f-e0690bf60cca"). InnerVolumeSpecName "kube-api-access-prldg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:16:08.697763 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:08.697735    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0537e8ab-6bd2-43df-b07f-e0690bf60cca-maas-api-xks-ca" pod "0537e8ab-6bd2-43df-b07f-e0690bf60cca" (UID: "0537e8ab-6bd2-43df-b07f-e0690bf60cca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:16:08.796114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:08.796017    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0537e8ab-6bd2-43df-b07f-e0690bf60cca-maas-api-openshift-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:16:08.796114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:08.796059    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prldg\" (UniqueName: \"kubernetes.io/projected/0537e8ab-6bd2-43df-b07f-e0690bf60cca-kube-api-access-prldg\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:16:08.796114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:08.796071    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0537e8ab-6bd2-43df-b07f-e0690bf60cca-maas-api-xks-ca\") on node \"ip-10-0-138-2.ec2.internal\" DevicePath \"\""
Oct 29 23:16:09.493603 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:09.493517    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24"
Oct 29 23:16:09.493603 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:09.493549    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-l4m24" event={"ID":"0537e8ab-6bd2-43df-b07f-e0690bf60cca","Type":"ContainerDied","Data":"28d47b9b6ffc64ad4ec73f4da0ae81c22ad4254011fa79576c389ad8a3255955"}
Oct 29 23:16:09.493603 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:09.493602    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28d47b9b6ffc64ad4ec73f4da0ae81c22ad4254011fa79576c389ad8a3255955"
Oct 29 23:16:13.693327 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:16:13.693286    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:16:15.891098 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:15.891058    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:16:20.897099 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:20.897059    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:16:38.382349 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:38.382253    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:16:43.693090 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:16:43.693051    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:16:47.494649 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.494605    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-7r58p"]
Oct 29 23:16:47.495324 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.495303    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0537e8ab-6bd2-43df-b07f-e0690bf60cca" containerName="revoke-keys"
Oct 29 23:16:47.495370 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.495328    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0537e8ab-6bd2-43df-b07f-e0690bf60cca" containerName="revoke-keys"
Oct 29 23:16:47.495432 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.495421    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0537e8ab-6bd2-43df-b07f-e0690bf60cca" containerName="revoke-keys"
Oct 29 23:16:47.499044 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.499016    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p"
Oct 29 23:16:47.519527 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.519479    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-7r58p"]
Oct 29 23:16:47.530114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.530080    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 29 23:16:47.530288 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.530086    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:16:47.530288 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.530086    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 23:16:47.530288 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.530086    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dzwhk\""
Oct 29 23:16:47.530288 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.530154    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:16:47.581160 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.581129    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49a4380c-4523-4e60-8e05-f2f926d2a382-openshift-service-ca\") pod \"maas-api-595cc648c-7r58p\" (UID: \"49a4380c-4523-4e60-8e05-f2f926d2a382\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p"
Oct 29 23:16:47.581352 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.581180    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49a4380c-4523-4e60-8e05-f2f926d2a382-metrics-certs\") pod \"maas-api-595cc648c-7r58p\" (UID: \"49a4380c-4523-4e60-8e05-f2f926d2a382\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p"
Oct 29 23:16:47.581352 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.581301    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49a4380c-4523-4e60-8e05-f2f926d2a382-maas-api-tls\") pod \"maas-api-595cc648c-7r58p\" (UID: \"49a4380c-4523-4e60-8e05-f2f926d2a382\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p"
Oct 29 23:16:47.581435 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.581336    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sjz7\" (UniqueName: \"kubernetes.io/projected/49a4380c-4523-4e60-8e05-f2f926d2a382-kube-api-access-6sjz7\") pod \"maas-api-595cc648c-7r58p\" (UID: \"49a4380c-4523-4e60-8e05-f2f926d2a382\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p"
Oct 29 23:16:47.682916 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.682876    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49a4380c-4523-4e60-8e05-f2f926d2a382-openshift-service-ca\") pod \"maas-api-595cc648c-7r58p\" (UID: \"49a4380c-4523-4e60-8e05-f2f926d2a382\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p"
Oct 29 23:16:47.683131 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.682936    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49a4380c-4523-4e60-8e05-f2f926d2a382-metrics-certs\") pod \"maas-api-595cc648c-7r58p\" (UID: \"49a4380c-4523-4e60-8e05-f2f926d2a382\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p"
Oct 29 23:16:47.683131 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.682990    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49a4380c-4523-4e60-8e05-f2f926d2a382-maas-api-tls\") pod \"maas-api-595cc648c-7r58p\" (UID: \"49a4380c-4523-4e60-8e05-f2f926d2a382\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p"
Oct 29 23:16:47.683131 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.683013    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sjz7\" (UniqueName: \"kubernetes.io/projected/49a4380c-4523-4e60-8e05-f2f926d2a382-kube-api-access-6sjz7\") pod \"maas-api-595cc648c-7r58p\" (UID: \"49a4380c-4523-4e60-8e05-f2f926d2a382\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p"
Oct 29 23:16:47.683598 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.683571    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49a4380c-4523-4e60-8e05-f2f926d2a382-openshift-service-ca\") pod \"maas-api-595cc648c-7r58p\" (UID: \"49a4380c-4523-4e60-8e05-f2f926d2a382\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p"
Oct 29 23:16:47.685653 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.685627    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49a4380c-4523-4e60-8e05-f2f926d2a382-metrics-certs\") pod \"maas-api-595cc648c-7r58p\" (UID: \"49a4380c-4523-4e60-8e05-f2f926d2a382\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p"
Oct 29 23:16:47.685751 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.685693    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49a4380c-4523-4e60-8e05-f2f926d2a382-maas-api-tls\") pod \"maas-api-595cc648c-7r58p\" (UID: \"49a4380c-4523-4e60-8e05-f2f926d2a382\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p"
Oct 29 23:16:47.702709 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.702678    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sjz7\" (UniqueName: \"kubernetes.io/projected/49a4380c-4523-4e60-8e05-f2f926d2a382-kube-api-access-6sjz7\") pod \"maas-api-595cc648c-7r58p\" (UID: \"49a4380c-4523-4e60-8e05-f2f926d2a382\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p"
Oct 29 23:16:47.812232 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:47.812142    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p"
Oct 29 23:16:48.118727 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:48.118700    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-7r58p"]
Oct 29 23:16:48.120723 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:16:48.120691    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a4380c_4523_4e60_8e05_f2f926d2a382.slice/crio-e1040d0b970dfa69e57d0bbd500a4a08ddc025fc2ebc0e4ba11d80cd3e7e4b57 WatchSource:0}: Error finding container e1040d0b970dfa69e57d0bbd500a4a08ddc025fc2ebc0e4ba11d80cd3e7e4b57: Status 404 returned error can't find the container with id e1040d0b970dfa69e57d0bbd500a4a08ddc025fc2ebc0e4ba11d80cd3e7e4b57
Oct 29 23:16:48.122065 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:48.122001    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:16:48.126694 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:48.126669    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:16:48.743923 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:48.743799    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p" event={"ID":"49a4380c-4523-4e60-8e05-f2f926d2a382","Type":"ContainerStarted","Data":"048c764939b2b89c026be624ef4bd73913183cd6fdfc89e428f8d55aaa14f5b8"}
Oct 29 23:16:48.743923 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:48.743888    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p" event={"ID":"49a4380c-4523-4e60-8e05-f2f926d2a382","Type":"ContainerStarted","Data":"e1040d0b970dfa69e57d0bbd500a4a08ddc025fc2ebc0e4ba11d80cd3e7e4b57"}
Oct 29 23:16:48.744547 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:48.744047    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p"
Oct 29 23:16:49.026592 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:49.026464    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p" podStartSLOduration=1.7560511399999998 podStartE2EDuration="2.026443052s" totalImagesPullingTime="270.391912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:48.787515841 +0000 UTC m=+3747.698959387" watchObservedRunningTime="2026-10-29 23:16:49.026443052 +0000 UTC m=+3747.937886610"
Oct 29 23:16:49.027733 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:49.027703    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"]
Oct 29 23:16:49.031389 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:49.031360    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7p87z"]
Oct 29 23:16:49.242467 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:49.242421    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef7d2cb2-e200-4c5b-82be-e778409fa251" path="/var/lib/kubelet/pods/ef7d2cb2-e200-4c5b-82be-e778409fa251/volumes"
Oct 29 23:16:54.758599 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:54.758555    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-7r58p"
Oct 29 23:16:56.181185 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:56.181144    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:16:59.496836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:59.495951    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:16:59.496836 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:16:59.496054    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g" podUID="e0384c7e-3969-4a0e-8e4d-7d9ffa718730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:17:09.987106 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:09.987055    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:17:13.887337 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:17:13.887294    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:17:14.499000 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:14.498957    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:17:14.499339 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:14.499305    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-kgx4g" podUID="e0384c7e-3969-4a0e-8e4d-7d9ffa718730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:17:15.081445 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:15.080942    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:17:15.081445 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:15.081016    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9g4hg" podUID="5716b06a-0198-4647-884c-3251a6a4b63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:17:37.593316 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:37.593272    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s75k9"]
Oct 29 23:17:43.694068 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:17:43.694022    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:17:48.982735 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:48.982699    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c5b76ff8b-rmfgt_4d3527ea-abd1-497c-a81e-531b46621769/manager/0.log"
Oct 29 23:17:50.345536 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:50.345506    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f7c578db4-jdgg4_d711fed7-3552-41ff-bca7-b5e79c526258/copy-manifests/0.log"
Oct 29 23:17:50.376818 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:50.376786    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f7c578db4-jdgg4_d711fed7-3552-41ff-bca7-b5e79c526258/manager/0.log"
Oct 29 23:17:51.326027 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:51.325994    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-7r58p_49a4380c-4523-4e60-8e05-f2f926d2a382/maas-api/0.log"
Oct 29 23:17:51.886687 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:51.886655    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-l4m24_0537e8ab-6bd2-43df-b07f-e0690bf60cca/revoke-keys/0.log"
Oct 29 23:17:52.087585 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:52.087551    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-4cmqc_1bf4495d-9d0f-4bef-a1a6-d9823cd4def5/revoke-keys/0.log"
Oct 29 23:17:55.088289 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:55.088258    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-th4wc_33b89a91-a78b-42e0-ac39-3d25c83d8844/manager/0.log"
Oct 29 23:17:55.278793 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:55.278756    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-f92wp_1ded2baf-62e1-43f1-8489-06e74ea61f5f/manager/0.log"
Oct 29 23:17:55.660990 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:55.660957    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-p4nbv_67df7b6e-f634-4469-8c4f-eaf2ca607afe/manager/0.log"
Oct 29 23:17:55.921330 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:55.921249    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-s75k9_81712066-dc50-44ed-a65f-820134d46bbd/limitador/0.log"
Oct 29 23:17:56.110341 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:56.110310    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-q9l85_6754baae-fae7-40dd-ab9b-54fd68603755/manager/0.log"
Oct 29 23:17:57.078090 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:57.078057    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bhpfs_f06f3791-4893-4cec-a5f3-9c0004bc2842/discovery/0.log"
Oct 29 23:17:57.261719 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:57.261659    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b84b87b7c-gk9x6_b8839f88-8053-4de1-b2d6-ecfbc851bb9c/kube-auth-proxy/0.log"
Oct 29 23:17:57.982038 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:57.982008    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-9g4hg_5716b06a-0198-4647-884c-3251a6a4b63e/istio-proxy/0.log"
Oct 29 23:17:58.347071 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:58.346983    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-kgx4g_e0384c7e-3969-4a0e-8e4d-7d9ffa718730/istio-proxy/0.log"
Oct 29 23:17:58.538348 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:58.538312    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-m55mt_d166a89c-98b6-4209-b8d8-5b405090031b/istio-proxy/0.log"
Oct 29 23:17:58.918713 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:17:58.918679    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-tm9cb_21f39456-b754-4dc5-bbca-9486af68dff5/istio-proxy/0.log"
Oct 29 23:18:01.000886 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:01.000851    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-rm75q_b2faf2e5-b0dd-4eb3-a396-ed3de57a737b/main/0.log"
Oct 29 23:18:01.362685 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:01.362599    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-5r7kn_e62d21f0-40a0-471a-9148-d5374553ee04/main/0.log"
Oct 29 23:18:01.749486 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:01.749446    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t_91680d4e-1b82-4456-96cb-0fa36ae5ff4f/main/0.log"
Oct 29 23:18:12.475763 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:12.475680    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c5b76ff8b-rmfgt_4d3527ea-abd1-497c-a81e-531b46621769/manager/0.log"
Oct 29 23:18:13.692384 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:18:13.692348    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:18:13.778841 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:13.778783    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f7c578db4-jdgg4_d711fed7-3552-41ff-bca7-b5e79c526258/copy-manifests/0.log"
Oct 29 23:18:13.807768 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:13.807736    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f7c578db4-jdgg4_d711fed7-3552-41ff-bca7-b5e79c526258/manager/0.log"
Oct 29 23:18:14.550503 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:14.550470    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-7r58p_49a4380c-4523-4e60-8e05-f2f926d2a382/maas-api/0.log"
Oct 29 23:18:15.084766 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:15.084723    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-l4m24_0537e8ab-6bd2-43df-b07f-e0690bf60cca/revoke-keys/0.log"
Oct 29 23:18:15.271524 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:15.271487    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-4cmqc_1bf4495d-9d0f-4bef-a1a6-d9823cd4def5/revoke-keys/0.log"
Oct 29 23:18:17.853034 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:17.853001    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-th4wc_33b89a91-a78b-42e0-ac39-3d25c83d8844/manager/0.log"
Oct 29 23:18:18.049950 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:18.049923    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-f92wp_1ded2baf-62e1-43f1-8489-06e74ea61f5f/manager/0.log"
Oct 29 23:18:18.417104 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:18.417070    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-p4nbv_67df7b6e-f634-4469-8c4f-eaf2ca607afe/manager/0.log"
Oct 29 23:18:18.669790 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:18.669707    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-s75k9_81712066-dc50-44ed-a65f-820134d46bbd/limitador/0.log"
Oct 29 23:18:18.845536 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:18.845500    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-q9l85_6754baae-fae7-40dd-ab9b-54fd68603755/manager/0.log"
Oct 29 23:18:19.584041 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:19.584008    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bhpfs_f06f3791-4893-4cec-a5f3-9c0004bc2842/discovery/0.log"
Oct 29 23:18:19.773794 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:19.773766    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b84b87b7c-gk9x6_b8839f88-8053-4de1-b2d6-ecfbc851bb9c/kube-auth-proxy/0.log"
Oct 29 23:18:20.550773 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:20.550744    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-9g4hg_5716b06a-0198-4647-884c-3251a6a4b63e/istio-proxy/0.log"
Oct 29 23:18:20.962493 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:20.962463    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-kgx4g_e0384c7e-3969-4a0e-8e4d-7d9ffa718730/istio-proxy/0.log"
Oct 29 23:18:21.173942 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:21.173907    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-m55mt_d166a89c-98b6-4209-b8d8-5b405090031b/istio-proxy/0.log"
Oct 29 23:18:21.552034 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:21.552002    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-tm9cb_21f39456-b754-4dc5-bbca-9486af68dff5/istio-proxy/0.log"
Oct 29 23:18:23.170303 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:23.170266    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-rm75q_b2faf2e5-b0dd-4eb3-a396-ed3de57a737b/main/0.log"
Oct 29 23:18:23.551251 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:23.551166    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-5r7kn_e62d21f0-40a0-471a-9148-d5374553ee04/main/0.log"
Oct 29 23:18:23.969730 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:23.969700    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-pbr4t_91680d4e-1b82-4456-96cb-0fa36ae5ff4f/main/0.log"
Oct 29 23:18:26.340660 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:26.340622    4559 scope.go:122] "RemoveContainer" containerID="cfe9742e10f3e243a965a27a029cb3a9abaacde226f5e71ec8abb54aa3c7c94e"
Oct 29 23:18:43.693600 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:18:43.693555    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:18:56.702208 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:56.702173    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c5b76ff8b-rmfgt_4d3527ea-abd1-497c-a81e-531b46621769/manager/0.log"
Oct 29 23:18:57.038412 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:18:57.038313    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c5b76ff8b-rmfgt_4d3527ea-abd1-497c-a81e-531b46621769/manager/0.log"
Oct 29 23:19:13.693530 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:19:13.693492    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:19:22.014848 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:22.014800    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 23:19:22.017435 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:22.017408    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlnh6_198638bc-ff2e-4c81-a599-616e7b015749/ovn-acl-logging/0.log"
Oct 29 23:19:39.215332 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:39.215249    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c5b76ff8b-rmfgt_4d3527ea-abd1-497c-a81e-531b46621769/manager/0.log"
Oct 29 23:19:39.480215 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:39.480132    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c5b76ff8b-rmfgt_4d3527ea-abd1-497c-a81e-531b46621769/manager/0.log"
Oct 29 23:19:43.692547 ip-10-0-138-2 kubenswrapper[4559]: E1029 23:19:43.692510    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dnx6b" containerName="extractor"
Oct 29 23:19:45.262201 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:45.262167    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p5bdf_02990faa-cba3-4132-9ca7-fe0f7202bbe5/global-pull-secret-syncer/0.log"
Oct 29 23:19:46.064414 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:46.064383    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v552v_1b6d3ad4-9eca-4ff5-913b-2eb405f4bb21/konnectivity-agent/0.log"
Oct 29 23:19:46.483154 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:46.483119    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-2.ec2.internal_c506ffe79e14398e87cdea51ca1d1f0f/haproxy/0.log"
Oct 29 23:19:52.912869 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:52.912833    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-th4wc_33b89a91-a78b-42e0-ac39-3d25c83d8844/manager/0.log"
Oct 29 23:19:53.048218 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:53.048187    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-f92wp_1ded2baf-62e1-43f1-8489-06e74ea61f5f/manager/0.log"
Oct 29 23:19:53.316205 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:53.316116    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-p4nbv_67df7b6e-f634-4469-8c4f-eaf2ca607afe/manager/0.log"
Oct 29 23:19:53.435999 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:53.435967    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-s75k9_81712066-dc50-44ed-a65f-820134d46bbd/limitador/0.log"
Oct 29 23:19:53.594269 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:53.594164    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-q9l85_6754baae-fae7-40dd-ab9b-54fd68603755/manager/0.log"
Oct 29 23:19:54.925114 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:54.925072    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_851cfdee-e076-421a-a269-c03c3fdaf5ce/alertmanager/0.log"
Oct 29 23:19:55.010914 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.010877    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"]
Oct 29 23:19:55.014672 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.014650    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.033369 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.033330    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mglbp\"/\"kube-root-ca.crt\""
Oct 29 23:19:55.035716 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.035690    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"]
Oct 29 23:19:55.046122 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.046094    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_851cfdee-e076-421a-a269-c03c3fdaf5ce/config-reloader/0.log"
Oct 29 23:19:55.046937 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.046917    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mglbp\"/\"openshift-service-ca.crt\""
Oct 29 23:19:55.047235 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.047216    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mglbp\"/\"default-dockercfg-d8vtb\""
Oct 29 23:19:55.066366 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.066326    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee-proc\") pod \"perf-node-gather-daemonset-gwtf6\" (UID: \"f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.066556 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.066387    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpbq4\" (UniqueName: \"kubernetes.io/projected/f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee-kube-api-access-vpbq4\") pod \"perf-node-gather-daemonset-gwtf6\" (UID: \"f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.066556 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.066485    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee-sys\") pod \"perf-node-gather-daemonset-gwtf6\" (UID: \"f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.066556 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.066548    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee-lib-modules\") pod \"perf-node-gather-daemonset-gwtf6\" (UID: \"f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.066773 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.066598    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee-podres\") pod \"perf-node-gather-daemonset-gwtf6\" (UID: \"f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.162653 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.162622    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_851cfdee-e076-421a-a269-c03c3fdaf5ce/kube-rbac-proxy-web/0.log"
Oct 29 23:19:55.168003 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.167979    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee-lib-modules\") pod \"perf-node-gather-daemonset-gwtf6\" (UID: \"f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.168079 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.168017    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee-podres\") pod \"perf-node-gather-daemonset-gwtf6\" (UID: \"f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.168079 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.168061    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee-proc\") pod \"perf-node-gather-daemonset-gwtf6\" (UID: \"f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.168164 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.168111    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpbq4\" (UniqueName: \"kubernetes.io/projected/f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee-kube-api-access-vpbq4\") pod \"perf-node-gather-daemonset-gwtf6\" (UID: \"f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.168164 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.168121    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee-lib-modules\") pod \"perf-node-gather-daemonset-gwtf6\" (UID: \"f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.168164 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.168147    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee-sys\") pod \"perf-node-gather-daemonset-gwtf6\" (UID: \"f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.168164 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.168153    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee-podres\") pod \"perf-node-gather-daemonset-gwtf6\" (UID: \"f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.168321 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.168171    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee-proc\") pod \"perf-node-gather-daemonset-gwtf6\" (UID: \"f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.168321 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.168231    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee-sys\") pod \"perf-node-gather-daemonset-gwtf6\" (UID: \"f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.188843 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.188745    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpbq4\" (UniqueName: \"kubernetes.io/projected/f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee-kube-api-access-vpbq4\") pod \"perf-node-gather-daemonset-gwtf6\" (UID: \"f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.278414 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.278383    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_851cfdee-e076-421a-a269-c03c3fdaf5ce/kube-rbac-proxy/0.log"
Oct 29 23:19:55.328705 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.328670    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.395024 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.394939    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_851cfdee-e076-421a-a269-c03c3fdaf5ce/kube-rbac-proxy-metric/0.log"
Oct 29 23:19:55.510261 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.510186    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_851cfdee-e076-421a-a269-c03c3fdaf5ce/prom-label-proxy/0.log"
Oct 29 23:19:55.630107 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.630086    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_851cfdee-e076-421a-a269-c03c3fdaf5ce/init-config-reloader/0.log"
Oct 29 23:19:55.630299 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.630275    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"]
Oct 29 23:19:55.632850 ip-10-0-138-2 kubenswrapper[4559]: W1029 23:19:55.632803    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf3d62bb7_9ac4_4a9d_bdea_5623cb7a43ee.slice/crio-071052f5a0c5fd50b0c6acf2fef25b84dd339d8d5ef616ab36c508cc99cbe952 WatchSource:0}: Error finding container 071052f5a0c5fd50b0c6acf2fef25b84dd339d8d5ef616ab36c508cc99cbe952: Status 404 returned error can't find the container with id 071052f5a0c5fd50b0c6acf2fef25b84dd339d8d5ef616ab36c508cc99cbe952
Oct 29 23:19:55.780079 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.779976    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6" event={"ID":"f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee","Type":"ContainerStarted","Data":"81505712d9f49c58f87264f9e46a4c33eb2eb6a87868823522c25a1d6de75f17"}
Oct 29 23:19:55.780079 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.780046    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:19:55.780079 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.780062    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6" event={"ID":"f3d62bb7-9ac4-4a9d-bdea-5623cb7a43ee","Type":"ContainerStarted","Data":"071052f5a0c5fd50b0c6acf2fef25b84dd339d8d5ef616ab36c508cc99cbe952"}
Oct 29 23:19:55.821433 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:55.821374    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6" podStartSLOduration=1.8213582590000001 podStartE2EDuration="1.821358259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:55.82070061 +0000 UTC m=+3934.732144168" watchObservedRunningTime="2026-10-29 23:19:55.821358259 +0000 UTC m=+3934.732801825"
Oct 29 23:19:56.217758 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:56.217730    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-58fdf66b86-ht9bn_21ba59b0-85ea-4547-be79-29c9a4008a3a/metrics-server/0.log"
Oct 29 23:19:56.799572 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:56.799530    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k45qw_cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a/node-exporter/0.log"
Oct 29 23:19:56.916262 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:56.916223    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k45qw_cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a/kube-rbac-proxy/0.log"
Oct 29 23:19:57.032714 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:57.032682    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k45qw_cf3d85d6-f7e5-47fd-9667-cbc0b566bf3a/init-textfile/0.log"
Oct 29 23:19:59.350138 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:59.350103    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c6cb5b5b-hnxcz_03c1223b-be7a-4231-96da-695de42bad71/thanos-query/0.log"
Oct 29 23:19:59.467498 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:59.467473    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c6cb5b5b-hnxcz_03c1223b-be7a-4231-96da-695de42bad71/kube-rbac-proxy-web/0.log"
Oct 29 23:19:59.584653 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:59.584614    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c6cb5b5b-hnxcz_03c1223b-be7a-4231-96da-695de42bad71/kube-rbac-proxy/0.log"
Oct 29 23:19:59.702211 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:59.702173    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c6cb5b5b-hnxcz_03c1223b-be7a-4231-96da-695de42bad71/prom-label-proxy/0.log"
Oct 29 23:19:59.818897 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:59.818870    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c6cb5b5b-hnxcz_03c1223b-be7a-4231-96da-695de42bad71/kube-rbac-proxy-rules/0.log"
Oct 29 23:19:59.935383 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:19:59.935356    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c6cb5b5b-hnxcz_03c1223b-be7a-4231-96da-695de42bad71/kube-rbac-proxy-metrics/0.log"
Oct 29 23:20:01.796091 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:20:01.796061    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-gwtf6"
Oct 29 23:20:03.500435 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:20:03.500399    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-78458cfc6-sx8jf_7b5f3cca-319e-4bd3-84de-f054f82473c2/console/0.log"
Oct 29 23:20:07.662103 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:20:07.662073    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xp49w_d4a55136-e7f7-46ac-b58e-24e95ab0aea8/dns/0.log"
Oct 29 23:20:07.775197 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:20:07.775170    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xp49w_d4a55136-e7f7-46ac-b58e-24e95ab0aea8/kube-rbac-proxy/0.log"
Oct 29 23:20:07.898708 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:20:07.898685    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jls5s_a66f6cc5-c6c7-476c-96e9-cd1fb85a78de/dns-node-resolver/0.log"
Oct 29 23:20:09.247063 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:20:09.247031    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-dcb998c6b-xlxth_70e57cd4-a6c4-4299-99bf-a29c21453751/registry/0.log"
Oct 29 23:20:09.462701 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:20:09.462666    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8bl7p_c51b17e1-8c9b-44cb-b0dc-aed1c4136a55/node-ca/0.log"
Oct 29 23:20:10.427570 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:20:10.427539    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bhpfs_f06f3791-4893-4cec-a5f3-9c0004bc2842/discovery/0.log"
Oct 29 23:20:10.458889 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:20:10.458861    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b84b87b7c-gk9x6_b8839f88-8053-4de1-b2d6-ecfbc851bb9c/kube-auth-proxy/0.log"
Oct 29 23:20:10.693713 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:20:10.693629    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-9g4hg_5716b06a-0198-4647-884c-3251a6a4b63e/istio-proxy/0.log"
Oct 29 23:20:10.809311 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:20:10.809279    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-kgx4g_e0384c7e-3969-4a0e-8e4d-7d9ffa718730/istio-proxy/0.log"
Oct 29 23:20:10.867951 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:20:10.867915    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-m55mt_d166a89c-98b6-4209-b8d8-5b405090031b/istio-proxy/0.log"
Oct 29 23:20:10.987044 ip-10-0-138-2 kubenswrapper[4559]: I1029 23:20:10.986960    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-tm9cb_21f39456-b754-4dc5-bbca-9486af68dff5/istio-proxy/0.log"